2024-03-27T02:46:13.461 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-27T02:46:13.462 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --verbose --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698/orig.config.yaml 2024-03-27T02:46:13.499 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T02:46:13.578 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622698' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCE2dRf1adqYcZW5xkSEOuI5DfuEsx1HXm8zZPbhMJqUbT0jV1f7DJMaWOlAr3wqgK+GdDh9ylXyGJ1Xn1vnopk= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFZ5kbULDe0hPvOIpHU+FY7udHbBIdr9lbE57zrSjNk6OqDJDxHa8jlUaa1AYKifJLnaUM/ut1vO0rQodJQVeaw= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-27T02:46:13.578 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-27T02:46:13.579 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-27T02:46:13.579 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-27T02:46:13.581 INFO:teuthology.task.internal:Checking packages... 2024-03-27T02:46:13.607 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-27T02:46:13.607 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-27T02:46:13.607 INFO:teuthology.packaging:ref: None 2024-03-27T02:46:13.607 INFO:teuthology.packaging:tag: None 2024-03-27T02:46:13.607 INFO:teuthology.packaging:branch: main 2024-03-27T02:46:13.607 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-27T02:46:13.608 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-27T02:46:13.863 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2471.g3c04e43e 2024-03-27T02:46:13.864 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-27T02:46:13.870 INFO:teuthology.task.internal:no buildpackages task found 2024-03-27T02:46:13.870 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-27T02:46:13.914 INFO:teuthology.task.internal:Saving configuration 2024-03-27T02:46:13.929 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-27T02:46:13.939 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T02:46:13.962 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 02:41:05.090063', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCE2dRf1adqYcZW5xkSEOuI5DfuEsx1HXm8zZPbhMJqUbT0jV1f7DJMaWOlAr3wqgK+GdDh9ylXyGJ1Xn1vnopk='} 2024-03-27T02:46:13.980 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 02:41:05.092037', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFZ5kbULDe0hPvOIpHU+FY7udHbBIdr9lbE57zrSjNk6OqDJDxHa8jlUaa1AYKifJLnaUM/ut1vO0rQodJQVeaw='} 2024-03-27T02:46:13.980 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-27T02:46:13.986 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-27T02:46:13.986 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-27T02:46:13.986 INFO:teuthology.run_tasks:Running task console_log... 2024-03-27T02:46:14.060 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4b4b4bc0d0>, signals=[15]) 2024-03-27T02:46:14.061 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-27T02:46:14.066 INFO:teuthology.task.internal:Opening connections... 2024-03-27T02:46:14.066 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-03-27T02:46:14.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T02:46:14.160 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-03-27T02:46:14.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T02:46:14.250 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-27T02:46:14.256 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-03-27T02:46:14.273 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-03-27T02:46:14.274 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-03-27T02:46:14.330 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-03-27T02:46:14.330 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-03-27T02:46:14.330 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-03-27T02:46:14.330 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-03-27T02:46:14.330 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-03-27T02:46:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-03-27T02:46:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-27T02:46:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-03-27T02:46:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-03-27T02:46:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-27T02:46:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-03-27T02:46:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-27T02:46:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-27T02:46:14.331 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-27T02:46:14.332 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-03-27T02:46:14.354 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-03-27T02:46:14.372 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-03-27T02:46:14.372 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-03-27T02:46:14.429 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-03-27T02:46:14.429 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-03-27T02:46:14.429 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-03-27T02:46:14.430 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-03-27T02:46:14.430 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-03-27T02:46:14.430 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-03-27T02:46:14.430 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-27T02:46:14.430 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-03-27T02:46:14.430 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-03-27T02:46:14.430 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-27T02:46:14.430 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-03-27T02:46:14.430 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-27T02:46:14.430 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-27T02:46:14.430 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-27T02:46:14.431 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-03-27T02:46:14.454 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-27T02:46:14.462 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-27T02:46:14.480 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-27T02:46:14.480 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-03-27T02:46:14.483 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-03-27T02:46:14.499 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-27T02:46:14.506 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-27T02:46:14.506 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-03-27T02:46:14.540 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-03-27T02:46:14.643 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-27T02:46:14.701 INFO:teuthology.run_tasks:Running task kernel... 2024-03-27T02:46:14.716 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-27T02:46:14.716 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-27T02:46:14.717 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-27T02:46:14.717 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-27T02:46:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T02:46:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T02:46:14.735 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-27T02:46:14.736 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-03-27T02:46:14.792 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-361.el9.x86_64 2024-03-27T02:46:14.792 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-361.el9.x86_64 2024-03-27T02:46:14.792 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-03-27T02:46:14.793 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-361.el9.x86_64 2024-03-27T02:46:14.793 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-361.el9.x86_64 2024-03-27T02:46:14.793 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-03-27T02:46:16.899 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-27T02:46:16.993 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-27T02:46:18.340 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-27T02:46:18.401 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-27T02:46:19.198 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-27T02:46:19.375 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-27T02:46:20.840 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-03-27T02:46:24.749 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-27T02:46:26.583 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-27T02:46:27.309 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 19 MB 00:07 2024-03-27T02:46:31.343 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-27T02:46:32.626 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 126 kB/s | 16 kB 00:00 2024-03-27T02:46:34.522 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-27T02:46:39.369 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-27T02:46:41.543 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-27T02:46:41.543 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-27T02:46:41.625 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T02:46:41.636 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T02:46:41.637 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-27T02:46:41.637 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T02:46:41.637 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T02:46:41.637 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-27T02:46:41.637 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-27T02:46:41.637 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-27T02:46:41.637 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-27T02:46:41.637 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-27T02:46:41.637 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T02:46:41.637 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T02:46:41.637 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T02:46:41.638 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-03-27T02:46:41.638 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T02:46:41.638 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 76 M 2024-03-27T02:46:41.638 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 123 M 2024-03-27T02:46:41.638 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T02:46:41.912 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 473 kB/s | 66 kB 00:00 2024-03-27T02:46:42.546 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-03-27T02:46:43.071 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 26 MB/s | 33 MB 00:01 2024-03-27T02:46:43.638 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 16 MB/s | 27 MB 00:01 2024-03-27T02:46:43.638 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:46:43.638 INFO:teuthology.orchestra.run.smithi153.stdout:Total 38 MB/s | 76 MB 00:02 2024-03-27T02:46:44.299 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T02:46:44.677 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T02:46:44.677 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T02:46:45.706 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T02:46:45.706 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T02:46:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T02:46:48.796 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-27T02:46:48.879 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T02:46:51.664 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T02:46:51.807 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T02:46:56.521 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T02:46:56.798 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T02:47:01.525 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T02:47:30.781 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T02:47:30.781 INFO:teuthology.orchestra.run.smithi153.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-27T02:47:30.782 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T02:47:30.825 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-27T02:47:31.171 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T02:47:31.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-27T02:47:31.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T02:47:31.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T02:47:31.606 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T02:47:31.606 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T02:47:31.606 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T02:47:31.606 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-27T02:47:31.606 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-27T02:47:31.606 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-27T02:47:31.607 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-27T02:47:31.607 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T02:47:31.607 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T02:47:31.925 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T02:47:32.021 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T02:47:32.022 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T02:47:32.022 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-27T02:47:32.022 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-27T02:47:32.022 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-27T02:47:32.022 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-27T02:47:32.022 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-03-27T02:47:32.022 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T02:47:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T02:47:32.102 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T02:47:32.102 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-03-27T02:47:32.140 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-03-27T02:47:32.140 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-03-27T02:47:32.140 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-03-27T02:47:32.140 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 68.el9 2024-03-27T02:47:32.140 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-03-27T02:47:32.140 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-27T02:47:32.140 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-03-27T02:47:32.140 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8264254 2024-03-27T02:47:32.140 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-03-27T02:47:32.140 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-27T02:47:32.140 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-27T02:47:32.140 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T02:47:32.141 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-03-27T02:47:32.142 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-27T02:47:32.142 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-432.el9.x86_64 2024-03-27T02:47:32.142 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-27T02:47:32.746 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-03-27T02:47:34.122 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-27T02:47:34.142 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-03-27T02:47:34.145 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-27T02:47:34.161 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.NUf38xHdmE 2024-03-27T02:47:34.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NUf38xHdmE 2024-03-27T02:47:34.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.NUf38xHdmE 2024-03-27T02:47:34.372 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.NUf38xHdmE is 6KB 2024-03-27T02:47:34.421 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.NUf38xHdmE 2024-03-27T02:47:34.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-03-27T02:47:34.499 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-27T02:47:34.500 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-27T02:47:34.500 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-27T02:47:34.500 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-27T02:47:34.501 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-27T02:47:34.678 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-03-27T02:48:04.681 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-27T02:48:04.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-27T02:48:04.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T02:48:23.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-27T02:48:32.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-27T02:48:32.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T02:48:35.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-27T02:48:47.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-27T02:48:47.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T02:49:05.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-27T02:49:20.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-27T02:49:20.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T02:49:21.112 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-27T02:49:21.569 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-27T02:49:21.569 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-27T02:49:21.569 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-27T02:49:21.583 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-432.el9.x86_64 2024-03-27T02:49:21.583 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-27T02:49:21.583 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-27T02:49:21.583 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-27T02:49:22.584 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-27T02:49:22.584 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-27T02:49:22.671 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-03-27T02:49:22.712 DEBUG:teuthology.parallel:result is None 2024-03-27T02:54:41.603 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 43 kB/s | 21 MB 08:14 2024-03-27T02:54:46.551 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-27T02:54:48.733 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-27T02:54:48.733 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-27T02:54:48.825 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T02:54:48.835 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T02:54:48.835 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-27T02:54:48.835 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T02:54:48.835 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T02:54:48.835 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-27T02:54:48.835 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-27T02:54:48.835 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-27T02:54:48.835 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-27T02:54:48.835 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-27T02:54:48.836 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T02:54:48.836 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T02:54:48.836 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T02:54:48.836 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-03-27T02:54:48.836 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T02:54:48.836 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 76 M 2024-03-27T02:54:48.837 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 123 M 2024-03-27T02:54:48.837 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T02:54:49.602 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 135 kB/s | 66 kB 00:00 2024-03-27T02:54:50.477 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 11 MB/s | 15 MB 00:01 2024-03-27T02:54:51.194 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 16 MB/s | 33 MB 00:02 2024-03-27T02:54:51.552 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 14 MB/s | 27 MB 00:01 2024-03-27T02:54:51.553 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:54:51.553 INFO:teuthology.orchestra.run.smithi196.stdout:Total 28 MB/s | 76 MB 00:02 2024-03-27T02:54:52.215 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T02:54:52.605 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T02:54:52.605 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T02:54:53.633 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T02:54:53.633 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T02:54:56.111 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T02:54:56.846 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-27T02:54:56.929 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T02:54:59.685 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T02:54:59.821 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T02:55:04.512 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T02:55:04.604 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T02:55:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T02:55:38.637 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T02:55:38.638 INFO:teuthology.orchestra.run.smithi196.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-27T02:55:38.638 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T02:55:38.681 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-27T02:55:39.066 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T02:55:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-27T02:55:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T02:55:39.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T02:55:39.477 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T02:55:39.477 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T02:55:39.478 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T02:55:39.478 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-27T02:55:39.478 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-27T02:55:39.478 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-27T02:55:39.478 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-27T02:55:39.478 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T02:55:39.478 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T02:55:39.799 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T02:55:39.897 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T02:55:39.897 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T02:55:39.898 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-27T02:55:39.898 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-27T02:55:39.898 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-27T02:55:39.898 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-27T02:55:39.898 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-03-27T02:55:39.898 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T02:55:39.977 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T02:55:39.978 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T02:55:39.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-03-27T02:55:40.012 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-03-27T02:55:40.012 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-03-27T02:55:40.012 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-03-27T02:55:40.012 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 68.el9 2024-03-27T02:55:40.012 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-03-27T02:55:40.012 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-27T02:55:40.012 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-03-27T02:55:40.012 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8264254 2024-03-27T02:55:40.012 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-03-27T02:55:40.012 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-27T02:55:40.012 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T02:55:40.013 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-03-27T02:55:40.014 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-27T02:55:40.014 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-432.el9.x86_64 2024-03-27T02:55:40.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-27T02:55:40.558 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-03-27T02:55:41.931 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-27T02:55:41.959 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-03-27T02:55:41.961 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-03-27T02:55:41.975 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.6PjAff4SfC 2024-03-27T02:55:41.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6PjAff4SfC 2024-03-27T02:55:42.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.6PjAff4SfC 2024-03-27T02:55:42.192 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.6PjAff4SfC is 6KB 2024-03-27T02:55:42.240 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.6PjAff4SfC 2024-03-27T02:55:42.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-03-27T02:55:42.318 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-27T02:55:42.319 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-27T02:55:42.319 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-27T02:55:42.319 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-27T02:55:42.320 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-27T02:55:42.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-03-27T02:56:12.477 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-27T02:56:12.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-27T02:56:12.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T02:56:31.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-03-27T02:56:40.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-27T02:56:40.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T02:57:40.012 DEBUG:teuthology.orchestra.remote:timed out 2024-03-27T02:57:52.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-27T02:57:52.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T02:57:52.611 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-03-27T02:57:52.991 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-27T02:57:52.991 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-27T02:57:52.991 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-03-27T02:57:53.007 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-432.el9.x86_64 2024-03-27T02:57:53.007 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-27T02:57:53.007 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-27T02:57:53.007 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-27T02:57:54.009 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-27T02:57:54.009 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-27T02:57:54.096 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-03-27T02:57:54.132 DEBUG:teuthology.parallel:result is None 2024-03-27T02:57:54.133 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-27T02:57:54.142 INFO:teuthology.task.internal:Creating test directory... 2024-03-27T02:57:54.142 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-27T02:57:54.147 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-27T02:57:54.165 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-27T02:57:54.175 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-27T02:57:54.194 INFO:teuthology.task.internal:Creating archive directory... 2024-03-27T02:57:54.194 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-27T02:57:54.206 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-27T02:57:54.242 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-27T02:57:54.249 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-27T02:57:54.249 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-27T02:57:54.277 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-27T02:57:54.317 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T02:57:54.322 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T02:57:54.331 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T02:57:54.336 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T02:57:54.337 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-27T02:57:54.345 INFO:teuthology.task.internal:Configuring sudo... 2024-03-27T02:57:54.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-27T02:57:54.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-27T02:57:54.403 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-27T02:57:54.416 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-27T02:57:54.416 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-27T02:57:54.420 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-27T02:57:54.458 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T02:57:54.516 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T02:57:54.568 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T02:57:54.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T02:57:54.693 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T02:57:54.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-27T02:57:54.761 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T02:57:54.804 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T02:57:54.860 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T02:57:54.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T02:57:54.987 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T02:57:54.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-27T02:57:55.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-03-27T02:57:55.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-03-27T02:57:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T02:57:55.142 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T02:57:55.604 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-27T02:57:55.612 INFO:teuthology.task.internal:Starting timer... 2024-03-27T02:57:55.612 INFO:teuthology.run_tasks:Running task pcp... 2024-03-27T02:57:55.654 INFO:teuthology.run_tasks:Running task selinux... 2024-03-27T02:57:55.678 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-03-27T02:57:55.732 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-03-27T02:57:55.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-03-27T02:57:55.777 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-03-27T02:57:55.779 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-27T02:57:55.780 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-03-27T02:57:55.805 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-03-27T02:57:55.805 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-03-27T02:57:55.846 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-03-27T02:57:55.847 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-03-27T02:57:55.847 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-27T02:57:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T02:57:55.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-27T02:57:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T02:57:55.914 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-27T02:57:55.914 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-03-27T02:57:55.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-03-27T02:57:55.988 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-27T02:57:55.999 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-27T02:57:56.004 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-27T02:57:56.138 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-27T02:57:56.155 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-27T02:57:56.156 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi153.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-27T03:02:28.300 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-03-27T03:02:28.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-27T03:02:28.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T03:02:28.412 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-27T03:02:28.489 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-27T03:02:28.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-27T03:02:28.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T03:02:28.586 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-03-27T03:02:28.668 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-27T03:02:28.668 INFO:teuthology.run_tasks:Running task clock... 2024-03-27T03:02:28.679 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-27T03:02:28.679 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-27T03:02:28.680 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T03:02:28.682 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-27T03:02:28.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T03:02:28.711 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-27T03:02:28.728 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-27T03:02:28.743 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-27T03:02:28.757 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-03-27T03:02:28.761 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-27T03:02:28.771 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-03-27T03:02:28.787 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-27T03:02:28.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-03-27T03:02:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-27T03:02:28.806 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-03-27T03:02:28.823 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-27T03:02:28.839 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-27T03:02:28.863 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-27T03:02:28.867 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T03:02:28.867 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-27T03:02:28.867 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T03:02:28.867 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T03:02:28.867 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T03:02:28.868 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T03:02:28.896 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-27T03:02:28.900 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T03:02:28.900 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-27T03:02:28.900 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T03:02:28.900 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T03:02:28.900 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T03:02:28.900 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T03:02:28.901 INFO:teuthology.run_tasks:Running task install... 2024-03-27T03:02:28.909 DEBUG:teuthology.task.install:project ceph 2024-03-27T03:02:28.909 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-27T03:02:28.909 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-27T03:02:28.909 INFO:teuthology.task.install:Using flavor: default 2024-03-27T03:02:28.915 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-27T03:02:28.915 INFO:teuthology.task.install:extra packages: [] 2024-03-27T03:02:28.915 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-27T03:02:28.916 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-27T03:02:28.918 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-27T03:02:28.918 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-27T03:02:29.065 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-27T03:02:29.065 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-27T03:02:29.149 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-27T03:02:29.149 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-27T03:02:29.254 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-27T03:02:29.254 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:02:29.254 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-27T03:02:29.283 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-27T03:02:29.283 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-27T03:02:29.333 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-27T03:02:29.333 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:02:29.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-27T03:02:29.351 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-27T03:02:29.362 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-27T03:02:29.362 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-27T03:02:29.432 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-27T03:02:29.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-27T03:02:29.497 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-03-27T03:02:29.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-03-27T03:02:29.512 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-27T03:02:29.578 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-03-27T03:02:29.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-03-27T03:02:29.744 INFO:teuthology.orchestra.run.smithi153.stdout:74 files removed 2024-03-27T03:02:29.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-03-27T03:02:29.823 INFO:teuthology.orchestra.run.smithi196.stdout:74 files removed 2024-03-27T03:02:29.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw 2024-03-27T03:02:30.541 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 210 kB/s | 82 kB 00:00 2024-03-27T03:02:30.598 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 155 kB/s | 82 kB 00:00 2024-03-27T03:02:30.966 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-27T03:02:31.040 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-27T03:02:31.333 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-03-27T03:02:31.383 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 6.3 kB/s | 1.9 kB 00:00 2024-03-27T03:02:31.785 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-27T03:02:31.827 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-27T03:02:33.277 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-27T03:02:33.694 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 2.6 MB/s | 2.0 MB 00:00 2024-03-27T03:02:35.009 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 8.3 MB/s | 8.0 MB 00:00 2024-03-27T03:02:35.137 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.0 MB 00:01 2024-03-27T03:02:36.852 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-27T03:02:38.019 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-03-27T03:02:43.103 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-03-27T03:02:44.745 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-27T03:02:45.880 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-27T03:02:49.118 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 715 kB/s | 6.1 MB 00:08 2024-03-27T03:02:50.761 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-27T03:02:50.872 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-27T03:02:52.462 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-27T03:02:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:02:53.342 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:02:53.343 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-27T03:02:53.343 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:02:53.343 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:02:53.343 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-27T03:02:53.343 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-03-27T03:02:53.343 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-27T03:02:53.343 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-27T03:02:53.343 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-27T03:02:53.343 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options 2024-03-27T03:02:53.343 INFO:teuthology.orchestra.run.smithi153.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-27T03:02:53.343 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-27T03:02:53.344 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-27T03:02:53.345 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-27T03:02:53.346 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-03-27T03:02:53.346 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-27T03:02:53.346 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:02:53.346 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:02:53.346 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:02:53.346 INFO:teuthology.orchestra.run.smithi153.stdout:Install 30 Packages 2024-03-27T03:02:53.346 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-03-27T03:02:53.346 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:02:53.346 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 58 M 2024-03-27T03:02:53.347 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:02:55.481 INFO:teuthology.orchestra.run.smithi153.stdout:(1/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 10 MB/s | 20 MB 00:01 2024-03-27T03:02:55.581 INFO:teuthology.orchestra.run.smithi153.stdout:(2/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.6 MB/s | 5.3 MB 00:02 2024-03-27T03:02:55.606 INFO:teuthology.orchestra.run.smithi153.stdout:(3/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 201 kB/s | 25 kB 00:00 2024-03-27T03:02:55.707 INFO:teuthology.orchestra.run.smithi153.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.9 MB/s | 498 kB 00:00 2024-03-27T03:02:55.765 INFO:teuthology.orchestra.run.smithi153.stdout:(5/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.9 MB/s | 742 kB 00:00 2024-03-27T03:02:55.857 INFO:teuthology.orchestra.run.smithi153.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 492 kB/s | 45 kB 00:00 2024-03-27T03:02:56.066 INFO:teuthology.orchestra.run.smithi153.stdout:(7/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 14 MB/s | 5.1 MB 00:00 2024-03-27T03:02:56.082 INFO:teuthology.orchestra.run.smithi153.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 583 kB/s | 131 kB 00:00 2024-03-27T03:02:56.191 INFO:teuthology.orchestra.run.smithi153.stdout:(9/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 3.6 MB/s | 9.7 MB 00:02 2024-03-27T03:02:56.216 INFO:teuthology.orchestra.run.smithi153.stdout:(10/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 1.1 MB/s | 162 kB 00:00 2024-03-27T03:02:56.242 INFO:teuthology.orchestra.run.smithi153.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.0 MB/s | 320 kB 00:00 2024-03-27T03:02:56.267 INFO:teuthology.orchestra.run.smithi153.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-27T03:02:56.317 INFO:teuthology.orchestra.run.smithi153.stdout:(13/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.3 MB/s | 298 kB 00:00 2024-03-27T03:02:56.401 INFO:teuthology.orchestra.run.smithi153.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 667 kB/s | 106 kB 00:00 2024-03-27T03:02:56.426 INFO:teuthology.orchestra.run.smithi153.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 539 kB/s | 44 kB 00:00 2024-03-27T03:02:56.460 INFO:teuthology.orchestra.run.smithi153.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 957 kB/s | 159 kB 00:00 2024-03-27T03:02:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 661 kB 00:00 2024-03-27T03:02:56.527 INFO:teuthology.orchestra.run.smithi153.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-03-27T03:02:56.552 INFO:teuthology.orchestra.run.smithi153.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 449 kB/s | 41 kB 00:00 2024-03-27T03:02:56.694 INFO:teuthology.orchestra.run.smithi153.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 254 kB/s | 36 kB 00:00 2024-03-27T03:02:56.719 INFO:teuthology.orchestra.run.smithi153.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 284 kB/s | 47 kB 00:00 2024-03-27T03:02:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 736 kB/s | 221 kB 00:00 2024-03-27T03:02:57.045 INFO:teuthology.orchestra.run.smithi153.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 107 kB/s | 26 kB 00:00 2024-03-27T03:02:57.220 INFO:teuthology.orchestra.run.smithi153.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 278 kB/s | 49 kB 00:00 2024-03-27T03:02:57.245 INFO:teuthology.orchestra.run.smithi153.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 560 kB/s | 309 kB 00:00 2024-03-27T03:02:57.396 INFO:teuthology.orchestra.run.smithi153.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 385 kB/s | 67 kB 00:00 2024-03-27T03:02:57.537 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-27T03:02:57.537 INFO:teuthology.orchestra.run.smithi153.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2024-03-27T03:02:57.638 INFO:teuthology.orchestra.run.smithi153.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 792 kB/s | 192 kB 00:00 2024-03-27T03:02:57.722 INFO:teuthology.orchestra.run.smithi153.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2024-03-27T03:02:58.014 INFO:teuthology.orchestra.run.smithi153.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-27T03:02:58.047 INFO:teuthology.orchestra.run.smithi153.stdout:(31/32): thrift-0.15.0-2.el9.x86_64.rpm 3.1 MB/s | 1.6 MB 00:00 2024-03-27T03:02:58.864 INFO:teuthology.orchestra.run.smithi153.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.8 MB/s | 3.4 MB 00:01 2024-03-27T03:02:58.867 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:02:58.868 INFO:teuthology.orchestra.run.smithi153.stdout:Total 10 MB/s | 58 MB 00:05 2024-03-27T03:02:58.903 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:02:58.942 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:02:58.942 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:02:59.431 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:02:59.431 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:02:59.983 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:02:59.992 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-27T03:02:59.993 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-27T03:02:59.994 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout:Installing weak dependencies: 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:02:59.995 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:02:59.996 INFO:teuthology.orchestra.run.smithi196.stdout:Install 30 Packages 2024-03-27T03:02:59.996 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 2 Packages 2024-03-27T03:02:59.997 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:02:59.997 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 58 M 2024-03-27T03:02:59.997 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:03:00.877 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:03:00.965 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-27T03:03:01.010 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-27T03:03:01.036 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-27T03:03:01.058 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-27T03:03:01.075 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-27T03:03:01.146 INFO:teuthology.orchestra.run.smithi196.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 9.6 MB/s | 9.7 MB 00:01 2024-03-27T03:03:01.196 INFO:teuthology.orchestra.run.smithi196.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 505 kB/s | 25 kB 00:00 2024-03-27T03:03:01.222 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-27T03:03:01.288 INFO:teuthology.orchestra.run.smithi196.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.9 MB/s | 742 kB 00:00 2024-03-27T03:03:01.295 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-27T03:03:01.338 INFO:teuthology.orchestra.run.smithi196.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 9.8 MB/s | 498 kB 00:00 2024-03-27T03:03:01.622 INFO:teuthology.orchestra.run.smithi196.stdout:(5/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.6 MB/s | 5.3 MB 00:01 2024-03-27T03:03:01.627 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-27T03:03:01.687 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-27T03:03:01.689 INFO:teuthology.orchestra.run.smithi196.stdout:(6/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 15 MB/s | 5.1 MB 00:00 2024-03-27T03:03:01.714 INFO:teuthology.orchestra.run.smithi196.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 491 kB/s | 45 kB 00:00 2024-03-27T03:03:01.739 INFO:teuthology.orchestra.run.smithi196.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.6 MB/s | 131 kB 00:00 2024-03-27T03:03:01.779 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-27T03:03:01.781 INFO:teuthology.orchestra.run.smithi196.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.4 MB/s | 162 kB 00:00 2024-03-27T03:03:01.806 INFO:teuthology.orchestra.run.smithi196.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 4.7 MB/s | 320 kB 00:00 2024-03-27T03:03:01.848 INFO:teuthology.orchestra.run.smithi196.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 298 kB 00:00 2024-03-27T03:03:01.865 INFO:teuthology.orchestra.run.smithi196.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-27T03:03:01.868 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-27T03:03:01.906 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-27T03:03:01.968 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-27T03:03:01.997 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-27T03:03:01.999 INFO:teuthology.orchestra.run.smithi196.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 704 kB/s | 106 kB 00:00 2024-03-27T03:03:02.024 INFO:teuthology.orchestra.run.smithi196.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-27T03:03:02.041 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-27T03:03:02.041 INFO:teuthology.orchestra.run.smithi196.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-27T03:03:02.108 INFO:teuthology.orchestra.run.smithi196.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-27T03:03:02.110 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-27T03:03:02.133 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-27T03:03:02.142 INFO:teuthology.orchestra.run.smithi196.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.7 MB/s | 282 kB 00:00 2024-03-27T03:03:02.159 INFO:teuthology.orchestra.run.smithi196.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 822 kB/s | 41 kB 00:00 2024-03-27T03:03:02.180 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-27T03:03:02.215 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-27T03:03:02.326 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-27T03:03:02.358 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-27T03:03:02.399 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-27T03:03:02.439 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-27T03:03:02.442 INFO:teuthology.orchestra.run.smithi196.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 127 kB/s | 36 kB 00:00 2024-03-27T03:03:02.468 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-27T03:03:02.500 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-27T03:03:02.559 INFO:teuthology.orchestra.run.smithi196.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 406 kB/s | 47 kB 00:00 2024-03-27T03:03:02.817 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-27T03:03:02.847 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-27T03:03:02.945 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-27T03:03:02.977 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-27T03:03:03.032 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-27T03:03:03.076 INFO:teuthology.orchestra.run.smithi196.stdout:(21/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 6.8 MB/s | 20 MB 00:02 2024-03-27T03:03:03.102 INFO:teuthology.orchestra.run.smithi196.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 230 kB/s | 221 kB 00:00 2024-03-27T03:03:03.343 INFO:teuthology.orchestra.run.smithi196.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 394 kB/s | 309 kB 00:00 2024-03-27T03:03:03.402 INFO:teuthology.orchestra.run.smithi196.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 86 kB/s | 26 kB 00:00 2024-03-27T03:03:03.461 INFO:teuthology.orchestra.run.smithi196.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-03-27T03:03:03.492 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-27T03:03:03.525 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-27T03:03:03.588 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-27T03:03:03.656 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-27T03:03:03.686 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-27T03:03:03.736 INFO:teuthology.orchestra.run.smithi196.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 202 kB/s | 67 kB 00:00 2024-03-27T03:03:03.811 INFO:teuthology.orchestra.run.smithi196.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.3 MB/s | 838 kB 00:00 2024-03-27T03:03:04.087 INFO:teuthology.orchestra.run.smithi196.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 547 kB/s | 192 kB 00:00 2024-03-27T03:03:04.137 INFO:teuthology.orchestra.run.smithi196.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 4.9 MB/s | 1.6 MB 00:00 2024-03-27T03:03:04.320 INFO:teuthology.orchestra.run.smithi196.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 4.4 MB 00:01 2024-03-27T03:03:04.612 INFO:teuthology.orchestra.run.smithi196.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 6.6 MB/s | 3.1 MB 00:00 2024-03-27T03:03:04.779 INFO:teuthology.orchestra.run.smithi196.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.9 MB/s | 3.4 MB 00:00 2024-03-27T03:03:04.784 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:03:04.784 INFO:teuthology.orchestra.run.smithi196.stdout:Total 12 MB/s | 58 MB 00:04 2024-03-27T03:03:04.818 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:03:04.856 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:03:04.856 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:03:05.368 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:03:05.369 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:03:05.883 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-27T03:03:05.924 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-27T03:03:06.479 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-27T03:03:06.510 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-27T03:03:06.606 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-27T03:03:06.606 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-27T03:03:06.607 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-27T03:03:06.607 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:06.645 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-27T03:03:06.917 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:03:07.004 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-27T03:03:07.034 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-27T03:03:07.059 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-27T03:03:07.082 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-27T03:03:07.099 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-27T03:03:07.274 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-27T03:03:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-27T03:03:07.645 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-27T03:03:07.694 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-27T03:03:07.768 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-27T03:03:07.855 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-27T03:03:07.887 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-27T03:03:07.950 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-27T03:03:07.970 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-27T03:03:08.005 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-27T03:03:08.072 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-27T03:03:08.097 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-27T03:03:08.161 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-27T03:03:08.196 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-27T03:03:08.302 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-27T03:03:08.339 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-27T03:03:08.379 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-27T03:03:08.403 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-27T03:03:08.432 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-27T03:03:08.465 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-27T03:03:08.780 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-27T03:03:08.812 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-27T03:03:08.900 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-27T03:03:08.933 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-27T03:03:08.974 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-27T03:03:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-27T03:03:09.456 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-27T03:03:09.510 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-27T03:03:09.578 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-27T03:03:09.608 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-27T03:03:11.770 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-27T03:03:11.821 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-27T03:03:12.377 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-27T03:03:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-27T03:03:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-27T03:03:12.507 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-27T03:03:12.507 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-27T03:03:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:12.551 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-27T03:03:16.272 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-27T03:03:16.272 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-03-27T03:03:16.272 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-03-27T03:03:16.272 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-03-27T03:03:16.272 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-03-27T03:03:16.272 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-03-27T03:03:16.272 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-03-27T03:03:16.272 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-03-27T03:03:16.272 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:17.203 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-27T03:03:17.230 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-27T03:03:17.231 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:03:17.231 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-27T03:03:17.231 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T03:03:17.231 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T03:03:17.231 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:17.232 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-27T03:03:17.322 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-27T03:03:17.322 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T03:03:18.434 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T03:03:18.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-27T03:03:18.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-27T03:03:18.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-27T03:03:18.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-27T03:03:18.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-27T03:03:18.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-27T03:03:18.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-27T03:03:18.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-27T03:03:18.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-27T03:03:18.435 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-27T03:03:18.436 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-27T03:03:18.436 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-27T03:03:18.436 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-27T03:03:18.436 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-27T03:03:18.436 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-27T03:03:18.436 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-27T03:03:18.436 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-27T03:03:18.436 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-27T03:03:18.436 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-27T03:03:19.549 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T03:03:19.549 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:19.549 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-03-27T03:03:19.549 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.549 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.549 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:03:19.549 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-27T03:03:19.549 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.549 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.549 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.549 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.549 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.550 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.551 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.551 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-27T03:03:19.551 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.551 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.551 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:19.551 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-27T03:03:19.551 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-27T03:03:19.551 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-27T03:03:19.551 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-03-27T03:03:19.551 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-27T03:03:19.551 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:19.551 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:03:19.870 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-03-27T03:03:20.422 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:30 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:03:20.578 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:03:20.578 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:03:20.578 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-27T03:03:20.578 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-03-27T03:03:20.579 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:20.580 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 48 M 2024-03-27T03:03:20.580 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 204 M 2024-03-27T03:03:20.580 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:03:21.038 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-27T03:03:21.096 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-27T03:03:21.130 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-27T03:03:21.255 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 408 kB/s | 163 kB 00:00 2024-03-27T03:03:21.280 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-27T03:03:22.197 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-27T03:03:22.197 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-03-27T03:03:22.197 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-03-27T03:03:22.197 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-03-27T03:03:22.197 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-03-27T03:03:22.197 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-03-27T03:03:22.197 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-03-27T03:03:22.197 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-03-27T03:03:22.197 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:23.130 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-27T03:03:23.154 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-27T03:03:23.154 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:03:23.154 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-27T03:03:23.154 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T03:03:23.155 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T03:03:23.155 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:23.156 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-27T03:03:23.223 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-27T03:03:23.223 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T03:03:23.605 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 17 MB/s | 47 MB 00:02 2024-03-27T03:03:23.608 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:03:23.609 INFO:teuthology.orchestra.run.smithi153.stdout:Total 16 MB/s | 48 MB 00:03 2024-03-27T03:03:23.623 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:03:23.644 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:03:23.645 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:03:24.024 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:03:24.025 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:03:24.333 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T03:03:24.333 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-27T03:03:24.333 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-27T03:03:24.333 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-27T03:03:24.333 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-27T03:03:24.333 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-27T03:03:24.334 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-27T03:03:24.335 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-27T03:03:24.335 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-27T03:03:24.335 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-27T03:03:24.335 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-27T03:03:24.335 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-27T03:03:24.335 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-27T03:03:24.335 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-27T03:03:24.335 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-27T03:03:24.336 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-27T03:03:24.336 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-27T03:03:24.336 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-27T03:03:24.336 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-27T03:03:24.336 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-27T03:03:24.336 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-27T03:03:24.336 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-27T03:03:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:03:24.664 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-27T03:03:24.704 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-27T03:03:24.745 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-27T03:03:24.799 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-27T03:03:24.825 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-27T03:03:25.275 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T03:03:25.275 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:25.275 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-03-27T03:03:25.275 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.276 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: re2-1:20211101-3.el9.x86_64 2024-03-27T03:03:25.277 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-27T03:03:25.278 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:25.278 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:03:25.656 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test 2024-03-27T03:03:26.221 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:29 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:03:26.380 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:03:26.381 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:03:26.381 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-03-27T03:03:26.382 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:26.383 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 48 M 2024-03-27T03:03:26.383 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 204 M 2024-03-27T03:03:26.383 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:03:26.845 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-27T03:03:26.904 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-27T03:03:26.930 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 597 kB/s | 163 kB 00:00 2024-03-27T03:03:26.955 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-27T03:03:27.163 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 801 kB/s | 187 kB 00:00 2024-03-27T03:03:28.989 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-27T03:03:28.992 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:03:28.993 INFO:teuthology.orchestra.run.smithi196.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-27T03:03:29.007 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:03:29.028 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:03:29.028 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:03:29.338 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-27T03:03:29.406 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:03:29.406 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:03:29.424 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-27T03:03:29.997 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:03:30.064 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-27T03:03:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-27T03:03:30.141 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-27T03:03:30.181 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-27T03:03:30.211 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-27T03:03:31.770 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-27T03:03:31.771 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-27T03:03:31.771 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-27T03:03:31.771 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-27T03:03:31.771 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-27T03:03:31.771 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-27T03:03:32.197 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-27T03:03:32.197 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:32.197 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:03:32.197 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:32.197 INFO:teuthology.orchestra.run.smithi153.stdout: jq-1.6-16.el9.x86_64 2024-03-27T03:03:32.197 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:32.197 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-27T03:03:32.197 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-27T03:03:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-27T03:03:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:32.198 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:03:32.479 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-03-27T03:03:33.071 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:43 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:03:33.238 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:03:33.240 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-03-27T03:03:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-27T03:03:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-27T03:03:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout:Install 36 Packages 2024-03-27T03:03:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:33.244 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 30 M 2024-03-27T03:03:33.244 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 107 M 2024-03-27T03:03:33.244 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:03:33.716 INFO:teuthology.orchestra.run.smithi153.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 21 kB/s | 6.6 kB 00:00 2024-03-27T03:03:34.058 INFO:teuthology.orchestra.run.smithi153.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 3.3 MB/s | 2.1 MB 00:00 2024-03-27T03:03:34.099 INFO:teuthology.orchestra.run.smithi153.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.1 MB/s | 1.5 MB 00:00 2024-03-27T03:03:34.225 INFO:teuthology.orchestra.run.smithi153.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.9 MB/s | 247 kB 00:00 2024-03-27T03:03:34.383 INFO:teuthology.orchestra.run.smithi153.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-27T03:03:34.434 INFO:teuthology.orchestra.run.smithi153.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-27T03:03:34.467 INFO:teuthology.orchestra.run.smithi153.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 954 kB/s | 32 kB 00:00 2024-03-27T03:03:34.501 INFO:teuthology.orchestra.run.smithi153.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-27T03:03:34.534 INFO:teuthology.orchestra.run.smithi153.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-27T03:03:34.602 INFO:teuthology.orchestra.run.smithi153.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-27T03:03:34.677 INFO:teuthology.orchestra.run.smithi153.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-27T03:03:34.719 INFO:teuthology.orchestra.run.smithi153.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-27T03:03:34.735 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-27T03:03:34.752 INFO:teuthology.orchestra.run.smithi153.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-27T03:03:34.794 INFO:teuthology.orchestra.run.smithi153.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-27T03:03:34.816 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-27T03:03:34.853 INFO:teuthology.orchestra.run.smithi153.stdout:(15/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 4.1 MB/s | 4.7 MB 00:01 2024-03-27T03:03:34.945 INFO:teuthology.orchestra.run.smithi153.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-27T03:03:35.053 INFO:teuthology.orchestra.run.smithi153.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 482 kB/s | 125 kB 00:00 2024-03-27T03:03:35.212 INFO:teuthology.orchestra.run.smithi153.stdout:(18/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 15 MB/s | 17 MB 00:01 2024-03-27T03:03:35.420 INFO:teuthology.orchestra.run.smithi153.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 118 kB/s | 43 kB 00:00 2024-03-27T03:03:35.462 INFO:teuthology.orchestra.run.smithi153.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 292 kB/s | 151 kB 00:00 2024-03-27T03:03:35.529 INFO:teuthology.orchestra.run.smithi153.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-27T03:03:35.596 INFO:teuthology.orchestra.run.smithi153.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 266 kB/s | 18 kB 00:00 2024-03-27T03:03:35.622 INFO:teuthology.orchestra.run.smithi153.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 447 kB/s | 172 kB 00:00 2024-03-27T03:03:35.722 INFO:teuthology.orchestra.run.smithi153.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-03-27T03:03:35.739 INFO:teuthology.orchestra.run.smithi153.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 163 kB/s | 23 kB 00:00 2024-03-27T03:03:35.756 INFO:teuthology.orchestra.run.smithi153.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 145 kB/s | 19 kB 00:00 2024-03-27T03:03:35.789 INFO:teuthology.orchestra.run.smithi153.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-03-27T03:03:35.839 INFO:teuthology.orchestra.run.smithi153.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 554 kB/s | 46 kB 00:00 2024-03-27T03:03:35.865 INFO:teuthology.orchestra.run.smithi153.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 725 kB/s | 79 kB 00:00 2024-03-27T03:03:35.890 INFO:teuthology.orchestra.run.smithi153.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-03-27T03:03:35.915 INFO:teuthology.orchestra.run.smithi153.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-03-27T03:03:35.949 INFO:teuthology.orchestra.run.smithi153.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-03-27T03:03:35.974 INFO:teuthology.orchestra.run.smithi153.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 427 kB/s | 36 kB 00:00 2024-03-27T03:03:36.041 INFO:teuthology.orchestra.run.smithi153.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-27T03:03:36.066 INFO:teuthology.orchestra.run.smithi153.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-03-27T03:03:36.116 INFO:teuthology.orchestra.run.smithi153.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.5 MB/s | 427 kB 00:00 2024-03-27T03:03:36.121 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:03:36.121 INFO:teuthology.orchestra.run.smithi153.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-27T03:03:36.145 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:03:36.186 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:03:36.186 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:03:36.598 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:03:36.598 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:03:37.150 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-27T03:03:37.151 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-27T03:03:37.151 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-27T03:03:37.151 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-27T03:03:37.151 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-27T03:03:37.151 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-27T03:03:37.476 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-27T03:03:37.476 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:37.477 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:03:37.477 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:37.477 INFO:teuthology.orchestra.run.smithi196.stdout: jq-1.6-16.el9.x86_64 2024-03-27T03:03:37.477 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:37.477 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-27T03:03:37.477 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-27T03:03:37.477 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-27T03:03:37.477 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:37.477 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:03:37.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph 2024-03-27T03:03:37.941 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:03:37.994 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-27T03:03:38.033 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-27T03:03:38.063 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-27T03:03:38.135 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-27T03:03:38.187 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-27T03:03:38.220 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-27T03:03:38.311 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-27T03:03:38.316 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:41 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:03:38.382 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-27T03:03:38.431 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-27T03:03:38.473 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:03:38.476 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-03-27T03:03:38.476 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-27T03:03:38.476 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-03-27T03:03:38.476 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:03:38.476 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-27T03:03:38.476 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-27T03:03:38.477 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-27T03:03:38.477 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-27T03:03:38.477 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-27T03:03:38.477 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-27T03:03:38.477 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-27T03:03:38.477 INFO:teuthology.orchestra.run.smithi196.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-27T03:03:38.477 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-27T03:03:38.477 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-27T03:03:38.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-27T03:03:38.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-27T03:03:38.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-27T03:03:38.477 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-27T03:03:38.478 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout:Install 36 Packages 2024-03-27T03:03:38.479 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:38.480 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 30 M 2024-03-27T03:03:38.480 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 107 M 2024-03-27T03:03:38.480 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:03:38.503 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-27T03:03:38.556 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-27T03:03:38.594 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-27T03:03:38.644 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-27T03:03:38.699 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-27T03:03:38.885 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-27T03:03:38.932 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-27T03:03:38.971 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-27T03:03:39.005 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-27T03:03:39.033 INFO:teuthology.orchestra.run.smithi196.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-27T03:03:39.035 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-27T03:03:39.113 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-27T03:03:39.145 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-27T03:03:39.175 INFO:teuthology.orchestra.run.smithi196.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-03-27T03:03:39.178 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-27T03:03:39.219 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-27T03:03:39.254 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-27T03:03:39.288 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-27T03:03:39.378 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-27T03:03:39.422 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-27T03:03:39.425 INFO:teuthology.orchestra.run.smithi196.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.4 MB/s | 1.5 MB 00:00 2024-03-27T03:03:39.492 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-27T03:03:39.517 INFO:teuthology.orchestra.run.smithi196.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.6 MB/s | 247 kB 00:00 2024-03-27T03:03:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-27T03:03:39.659 INFO:teuthology.orchestra.run.smithi196.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-27T03:03:39.709 INFO:teuthology.orchestra.run.smithi196.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-27T03:03:39.751 INFO:teuthology.orchestra.run.smithi196.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 760 kB/s | 32 kB 00:00 2024-03-27T03:03:39.753 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-27T03:03:39.793 INFO:teuthology.orchestra.run.smithi196.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-27T03:03:39.835 INFO:teuthology.orchestra.run.smithi196.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 926 kB/s | 39 kB 00:00 2024-03-27T03:03:39.920 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-27T03:03:39.949 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-27T03:03:39.974 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-27T03:03:39.974 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:03:39.975 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-27T03:03:39.975 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T03:03:39.975 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T03:03:39.975 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:40.002 INFO:teuthology.orchestra.run.smithi196.stdout:(10/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 20 MB/s | 17 MB 00:00 2024-03-27T03:03:40.028 INFO:teuthology.orchestra.run.smithi196.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 241 kB 00:00 2024-03-27T03:03:40.103 INFO:teuthology.orchestra.run.smithi196.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-27T03:03:40.138 INFO:teuthology.orchestra.run.smithi196.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 950 kB/s | 103 kB 00:00 2024-03-27T03:03:40.162 INFO:teuthology.orchestra.run.smithi196.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-27T03:03:40.187 INFO:teuthology.orchestra.run.smithi196.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-27T03:03:40.254 INFO:teuthology.orchestra.run.smithi196.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-03-27T03:03:40.362 INFO:teuthology.orchestra.run.smithi196.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-03-27T03:03:40.605 INFO:teuthology.orchestra.run.smithi196.stdout:(18/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.0 MB/s | 4.7 MB 00:01 2024-03-27T03:03:40.646 INFO:teuthology.orchestra.run.smithi196.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 153 kB/s | 43 kB 00:00 2024-03-27T03:03:40.705 INFO:teuthology.orchestra.run.smithi196.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 335 kB/s | 151 kB 00:00 2024-03-27T03:03:40.772 INFO:teuthology.orchestra.run.smithi196.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-27T03:03:40.855 INFO:teuthology.orchestra.run.smithi196.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 212 kB/s | 18 kB 00:00 2024-03-27T03:03:40.906 INFO:teuthology.orchestra.run.smithi196.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-27T03:03:40.923 INFO:teuthology.orchestra.run.smithi196.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-03-27T03:03:40.973 INFO:teuthology.orchestra.run.smithi196.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-27T03:03:40.998 INFO:teuthology.orchestra.run.smithi196.stdout:(26/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 437 kB/s | 172 kB 00:00 2024-03-27T03:03:41.015 INFO:teuthology.orchestra.run.smithi196.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2024-03-27T03:03:41.041 INFO:teuthology.orchestra.run.smithi196.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 689 kB/s | 46 kB 00:00 2024-03-27T03:03:41.082 INFO:teuthology.orchestra.run.smithi196.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 940 kB/s | 79 kB 00:00 2024-03-27T03:03:41.108 INFO:teuthology.orchestra.run.smithi196.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-03-27T03:03:41.174 INFO:teuthology.orchestra.run.smithi196.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 978 kB/s | 90 kB 00:00 2024-03-27T03:03:41.200 INFO:teuthology.orchestra.run.smithi196.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-27T03:03:41.225 INFO:teuthology.orchestra.run.smithi196.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 306 kB/s | 36 kB 00:00 2024-03-27T03:03:41.292 INFO:teuthology.orchestra.run.smithi196.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-27T03:03:41.317 INFO:teuthology.orchestra.run.smithi196.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-27T03:03:41.359 INFO:teuthology.orchestra.run.smithi196.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-27T03:03:41.365 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:03:41.365 INFO:teuthology.orchestra.run.smithi196.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-27T03:03:41.387 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:03:41.416 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-27T03:03:41.426 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:03:41.426 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:03:41.442 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-27T03:03:41.442 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:03:41.442 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-27T03:03:41.443 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T03:03:41.443 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T03:03:41.443 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:41.839 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:03:41.839 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:03:41.883 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-27T03:03:41.907 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-27T03:03:41.907 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:03:41.907 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-27T03:03:41.907 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T03:03:41.907 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T03:03:41.907 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:42.126 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-27T03:03:42.149 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-27T03:03:42.149 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:03:42.149 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-27T03:03:42.149 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T03:03:42.149 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T03:03:42.149 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:42.215 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-27T03:03:43.199 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:03:43.262 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-27T03:03:43.295 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-27T03:03:43.322 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-27T03:03:43.395 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-27T03:03:43.446 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-27T03:03:43.479 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-27T03:03:43.570 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-27T03:03:43.657 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-27T03:03:43.706 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-27T03:03:43.778 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-27T03:03:43.832 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-27T03:03:43.870 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-27T03:03:43.919 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-27T03:03:43.978 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-27T03:03:43.979 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-27T03:03:43.979 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-27T03:03:43.980 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-27T03:03:43.980 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-27T03:03:43.980 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-27T03:03:43.980 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-27T03:03:43.980 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-27T03:03:43.980 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-27T03:03:43.980 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-27T03:03:43.981 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-27T03:03:43.981 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-27T03:03:43.981 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-27T03:03:43.981 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-27T03:03:43.981 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-27T03:03:43.981 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-27T03:03:43.981 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-27T03:03:43.981 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-27T03:03:43.981 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-27T03:03:43.981 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-27T03:03:43.981 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-27T03:03:43.982 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-27T03:03:43.982 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-27T03:03:43.982 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-27T03:03:43.982 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-27T03:03:43.982 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-27T03:03:43.982 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-27T03:03:43.982 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-27T03:03:43.982 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-27T03:03:43.982 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-27T03:03:43.982 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-27T03:03:43.982 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-27T03:03:43.983 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-27T03:03:43.983 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-27T03:03:43.983 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-27T03:03:43.983 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-27T03:03:43.983 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-27T03:03:44.158 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-27T03:03:44.208 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-27T03:03:44.238 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-27T03:03:44.273 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-27T03:03:44.303 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-27T03:03:44.379 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-27T03:03:44.414 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-27T03:03:44.447 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-27T03:03:44.487 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-27T03:03:44.522 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-27T03:03:44.565 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-27T03:03:44.646 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-27T03:03:44.688 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-27T03:03:44.759 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-27T03:03:44.844 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-27T03:03:44.845 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-03-27T03:03:44.846 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-27T03:03:44.847 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-27T03:03:44.847 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-27T03:03:44.847 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-27T03:03:44.847 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-27T03:03:44.847 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-27T03:03:44.847 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-27T03:03:44.847 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-27T03:03:44.847 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-27T03:03:44.847 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-27T03:03:44.847 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-27T03:03:44.847 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:44.847 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:03:44.896 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-27T03:03:45.012 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-27T03:03:45.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-03-27T03:03:45.162 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-27T03:03:45.185 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-27T03:03:45.209 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-27T03:03:45.210 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:03:45.210 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-27T03:03:45.210 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T03:03:45.210 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T03:03:45.210 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:45.688 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:55 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:03:45.785 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:03:45.837 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:03:45.838 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-27T03:03:45.838 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:03:45.907 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-03-27T03:03:46.469 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:56 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:03:46.620 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:03:46.620 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:03:46.620 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-27T03:03:46.621 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:03:46.621 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:03:46.621 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-27T03:03:46.621 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:46.621 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:03:46.621 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:03:46.621 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-27T03:03:46.621 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:46.621 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 747 k 2024-03-27T03:03:46.621 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 754 k 2024-03-27T03:03:46.621 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:03:46.652 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-27T03:03:46.679 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-27T03:03:46.679 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:03:46.679 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-27T03:03:46.679 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T03:03:46.679 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T03:03:46.679 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:46.940 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.3 MB/s | 747 kB 00:00 2024-03-27T03:03:46.940 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:03:46.941 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.3 MB/s | 747 kB 00:00 2024-03-27T03:03:46.941 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:03:46.946 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:03:46.946 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:03:46.952 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:03:46.952 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:03:47.027 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:03:47.136 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-27T03:03:47.159 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-27T03:03:47.159 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:03:47.159 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-27T03:03:47.159 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T03:03:47.160 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T03:03:47.160 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-27T03:03:47.394 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-27T03:03:47.394 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:03:47.394 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-27T03:03:47.394 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T03:03:47.394 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T03:03:47.394 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-27T03:03:47.481 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-27T03:03:47.560 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-27T03:03:47.934 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-27T03:03:48.219 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-27T03:03:48.219 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:48.219 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:03:48.219 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:03:48.219 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:48.219 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:03:48.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-03-27T03:03:49.002 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:59 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:03:49.154 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-27T03:03:49.154 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-27T03:03:49.155 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-27T03:03:49.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-27T03:03:49.157 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-27T03:03:49.157 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-27T03:03:49.157 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-27T03:03:49.157 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-27T03:03:49.157 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-27T03:03:49.157 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-27T03:03:49.158 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:03:49.158 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:03:49.158 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-27T03:03:49.158 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:03:49.158 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:03:49.158 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-27T03:03:49.158 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:49.158 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:03:49.158 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:03:49.158 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-27T03:03:49.158 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:49.158 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 145 k 2024-03-27T03:03:49.159 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 436 k 2024-03-27T03:03:49.159 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:03:49.379 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 655 kB/s | 145 kB 00:00 2024-03-27T03:03:49.380 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:03:49.380 INFO:teuthology.orchestra.run.smithi153.stdout:Total 651 kB/s | 145 kB 00:00 2024-03-27T03:03:49.380 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:03:49.387 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:03:49.387 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:03:49.439 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:03:49.439 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:03:49.590 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:03:49.655 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-27T03:03:49.679 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-27T03:03:49.679 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:03:49.679 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-27T03:03:49.679 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:50.076 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-27T03:03:50.076 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:50.076 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:03:50.076 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: lua-5.4.4-4.el9.x86_64 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-27T03:03:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-27T03:03:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-27T03:03:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-27T03:03:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-27T03:03:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-27T03:03:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-27T03:03:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-27T03:03:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-27T03:03:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:50.079 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:03:50.403 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base 2024-03-27T03:03:50.660 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-27T03:03:50.660 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:50.660 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:03:50.660 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:50.661 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:50.661 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:03:50.916 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-03-27T03:03:50.963 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:53 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:03:51.061 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:03:51.112 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:03:51.113 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-27T03:03:51.113 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:03:51.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install cephadm 2024-03-27T03:03:51.487 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:01 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:03:51.585 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:03:51.639 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:03:51.640 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-27T03:03:51.640 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:03:51.708 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-03-27T03:03:51.733 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:54 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:03:51.883 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:03:51.883 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:03:51.883 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-27T03:03:51.883 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:03:51.884 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:03:51.884 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-27T03:03:51.884 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:51.884 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:03:51.884 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:03:51.884 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-27T03:03:51.884 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:51.885 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 747 k 2024-03-27T03:03:51.885 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 754 k 2024-03-27T03:03:51.885 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:03:52.161 INFO:teuthology.orchestra.run.smithi196.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.6 MB/s | 747 kB 00:00 2024-03-27T03:03:52.162 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:03:52.162 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.6 MB/s | 747 kB 00:00 2024-03-27T03:03:52.162 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:03:52.167 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:03:52.167 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:03:52.173 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:03:52.173 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:03:52.245 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:03:52.267 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:02 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:03:52.417 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:03:52.418 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-03-27T03:03:52.418 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-27T03:03:52.418 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-03-27T03:03:52.418 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:03:52.418 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-27T03:03:52.419 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:52.420 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:03:52.420 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-03-27T03:03:52.420 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-03-27T03:03:52.420 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:52.420 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 6.9 M 2024-03-27T03:03:52.420 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-03-27T03:03:52.420 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:03:52.637 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-27T03:03:52.725 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-27T03:03:52.924 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 157 kB/s | 31 kB 00:00 2024-03-27T03:03:53.058 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 46 kB/s | 15 kB 00:00 2024-03-27T03:03:53.082 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-27T03:03:53.099 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.8 MB/s | 1.8 MB 00:00 2024-03-27T03:03:53.158 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-27T03:03:53.183 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-27T03:03:53.339 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-27T03:03:53.339 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:53.339 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:03:53.339 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:03:53.339 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:53.339 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:03:53.459 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-03-27T03:03:53.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache 2024-03-27T03:03:53.592 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 48 kB/s | 19 kB 00:00 2024-03-27T03:03:53.926 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 714 kB/s | 548 kB 00:00 2024-03-27T03:03:53.951 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 403 kB/s | 144 kB 00:00 2024-03-27T03:03:54.034 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 283 kB/s | 31 kB 00:00 2024-03-27T03:03:54.124 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:57 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:03:54.151 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 941 kB/s | 188 kB 00:00 2024-03-27T03:03:54.278 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:03:54.279 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:03:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-27T03:03:54.279 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:03:54.279 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:03:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-27T03:03:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:54.279 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:03:54.279 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:03:54.279 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-27T03:03:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:54.280 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 145 k 2024-03-27T03:03:54.280 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 436 k 2024-03-27T03:03:54.280 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:03:54.410 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.1 MB/s | 2.0 MB 00:00 2024-03-27T03:03:54.415 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:03:54.415 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.4 MB/s | 6.9 MB 00:01 2024-03-27T03:03:54.433 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:03:54.450 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:03:54.450 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:03:54.634 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:03:54.634 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:03:54.636 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 408 kB/s | 145 kB 00:00 2024-03-27T03:03:54.636 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:03:54.637 INFO:teuthology.orchestra.run.smithi196.stdout:Total 407 kB/s | 145 kB 00:00 2024-03-27T03:03:54.637 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:03:54.645 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:03:54.645 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:03:54.698 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:03:54.698 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:03:54.842 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:03:54.912 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-27T03:03:54.936 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-27T03:03:54.937 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:03:54.937 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-27T03:03:54.937 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:55.142 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:03:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-27T03:03:55.271 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-27T03:03:55.318 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-27T03:03:55.356 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-27T03:03:55.458 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-27T03:03:55.591 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-27T03:03:55.691 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-27T03:03:55.735 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-27T03:03:55.789 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-27T03:03:55.821 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-27T03:03:55.854 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-27T03:03:55.855 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:55.855 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:03:55.855 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:03:55.855 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:55.855 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:03:56.063 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-27T03:03:56.096 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-27T03:03:56.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr 2024-03-27T03:03:56.667 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:59 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:03:56.764 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:03:56.815 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:03:56.816 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-27T03:03:56.816 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:03:56.863 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-27T03:03:56.863 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-27T03:03:56.863 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-27T03:03:56.863 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-27T03:03:56.863 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-27T03:03:56.863 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-27T03:03:56.863 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-27T03:03:56.863 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-27T03:03:56.863 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-27T03:03:56.864 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-27T03:03:56.864 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-27T03:03:56.864 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-27T03:03:56.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-dashboard 2024-03-27T03:03:57.210 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-27T03:03:57.210 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:03:57.210 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-27T03:03:57.210 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:03:57.211 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:03:57.211 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:03:57.211 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-27T03:03:57.211 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-27T03:03:57.211 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-27T03:03:57.211 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-27T03:03:57.211 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-27T03:03:57.211 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-27T03:03:57.211 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-27T03:03:57.211 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-27T03:03:57.211 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:57.211 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:03:57.426 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:00 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:03:57.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-27T03:03:57.573 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:03:57.574 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-03-27T03:03:57.574 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-27T03:03:57.574 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-03-27T03:03:57.574 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:03:57.574 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-27T03:03:57.574 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-27T03:03:57.574 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-27T03:03:57.574 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-27T03:03:57.574 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-27T03:03:57.574 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-27T03:03:57.574 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-27T03:03:57.575 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-27T03:03:57.575 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-27T03:03:57.575 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-27T03:03:57.575 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-27T03:03:57.575 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-27T03:03:57.575 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-27T03:03:57.575 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:57.575 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:03:57.575 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-03-27T03:03:57.575 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-03-27T03:03:57.575 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:03:57.576 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 6.9 M 2024-03-27T03:03:57.576 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-03-27T03:03:57.576 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:03:58.024 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:08 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:03:58.057 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 56 kB/s | 15 kB 00:00 2024-03-27T03:03:58.082 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 105 kB/s | 31 kB 00:00 2024-03-27T03:03:58.175 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:03:58.176 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-27T03:03:58.176 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-27T03:03:58.177 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-27T03:03:58.178 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:58.178 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:03:58.178 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-27T03:03:58.178 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-03-27T03:03:58.178 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:03:58.178 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 37 M 2024-03-27T03:03:58.179 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 209 M 2024-03-27T03:03:58.179 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:03:58.183 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.5 MB/s | 1.8 MB 00:00 2024-03-27T03:03:58.283 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 238 kB 00:00 2024-03-27T03:03:58.400 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-03-27T03:03:58.456 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-03-27T03:03:58.659 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-27T03:03:58.791 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 208 kB/s | 32 kB 00:00 2024-03-27T03:03:58.826 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-03-27T03:03:58.833 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 416 kB/s | 17 kB 00:00 2024-03-27T03:03:58.884 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 641 kB/s | 144 kB 00:00 2024-03-27T03:03:58.908 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-27T03:03:58.909 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 368 kB/s | 31 kB 00:00 2024-03-27T03:03:58.933 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 345 kB/s | 34 kB 00:00 2024-03-27T03:03:59.027 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-03-27T03:03:59.076 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-27T03:03:59.127 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.0 MB/s | 2.0 MB 00:00 2024-03-27T03:03:59.132 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:03:59.132 INFO:teuthology.orchestra.run.smithi196.stdout:Total 4.4 MB/s | 6.9 MB 00:01 2024-03-27T03:03:59.134 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-27T03:03:59.149 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:03:59.159 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-27T03:03:59.165 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:03:59.165 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:03:59.349 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:03:59.350 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:03:59.385 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 63 MB/s | 16 MB 00:00 2024-03-27T03:03:59.477 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 8.8 MB/s | 7.4 MB 00:00 2024-03-27T03:03:59.710 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 600 kB/s | 195 kB 00:00 2024-03-27T03:03:59.744 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-27T03:03:59.769 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 840 kB/s | 245 kB 00:00 2024-03-27T03:03:59.771 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:03:59.771 INFO:teuthology.orchestra.run.smithi153.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-27T03:03:59.812 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:03:59.837 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:03:59.837 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:03:59.865 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:03:59.943 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-27T03:03:59.987 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-27T03:04:00.042 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-27T03:04:00.073 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-27T03:04:00.177 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-27T03:04:00.203 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:04:00.203 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:04:00.305 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-27T03:04:00.391 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-27T03:04:00.435 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-27T03:04:00.465 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-27T03:04:00.496 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-27T03:04:00.746 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-27T03:04:00.788 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-27T03:04:00.937 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:04:01.008 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-27T03:04:01.040 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-27T03:04:01.133 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-27T03:04:01.176 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-27T03:04:01.355 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T03:04:01.384 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T03:04:01.461 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-27T03:04:01.608 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-27T03:04:01.608 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-27T03:04:01.609 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-27T03:04:01.609 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-27T03:04:01.609 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-27T03:04:01.609 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-27T03:04:01.609 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-27T03:04:01.609 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-27T03:04:01.609 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-27T03:04:01.609 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-27T03:04:01.609 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-27T03:04:01.609 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-27T03:04:01.958 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-27T03:04:01.958 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:01.958 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:04:01.958 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-27T03:04:01.958 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:04:01.958 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:04:01.958 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:04:01.958 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-27T03:04:01.958 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-27T03:04:01.958 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-27T03:04:01.959 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-27T03:04:01.959 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-27T03:04:01.959 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-27T03:04:01.959 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-27T03:04:01.959 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-27T03:04:01.959 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:01.959 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:02.132 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-27T03:04:02.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-27T03:04:02.237 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-27T03:04:02.763 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:05 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:02.914 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:02.915 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-27T03:04:02.915 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-27T03:04:02.915 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-27T03:04:02.915 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:04:02.915 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:04:02.916 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-27T03:04:02.917 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-03-27T03:04:02.917 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:02.917 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 37 M 2024-03-27T03:04:02.917 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 209 M 2024-03-27T03:04:02.917 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:04:03.480 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-27T03:04:03.575 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 207 kB/s | 32 kB 00:00 2024-03-27T03:04:03.617 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 422 kB/s | 17 kB 00:00 2024-03-27T03:04:03.659 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 844 kB/s | 34 kB 00:00 2024-03-27T03:04:03.710 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-27T03:04:03.868 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-27T03:04:03.927 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2024-03-27T03:04:03.960 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-27T03:04:04.169 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 65 MB/s | 16 MB 00:00 2024-03-27T03:04:04.286 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-03-27T03:04:04.311 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-27T03:04:04.516 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-27T03:04:04.549 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-27T03:04:04.561 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 891 kB/s | 245 kB 00:00 2024-03-27T03:04:07.071 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-27T03:04:07.071 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-27T03:04:07.071 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-27T03:04:07.071 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-27T03:04:07.071 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-27T03:04:07.071 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T03:04:07.071 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T03:04:07.071 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-27T03:04:07.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-27T03:04:07.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-27T03:04:07.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-27T03:04:07.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-27T03:04:07.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-27T03:04:07.552 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:07.552 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:04:07.552 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:04:07.552 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-27T03:04:07.552 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-27T03:04:07.552 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-27T03:04:07.553 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-27T03:04:07.553 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-27T03:04:07.553 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-27T03:04:07.553 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-27T03:04:07.553 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-27T03:04:07.553 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-27T03:04:07.553 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-27T03:04:07.553 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-27T03:04:07.553 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:07.553 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:07.917 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-03-27T03:04:08.262 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.5 MB/s | 7.4 MB 00:04 2024-03-27T03:04:08.265 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:08.265 INFO:teuthology.orchestra.run.smithi196.stdout:Total 6.9 MB/s | 37 MB 00:05 2024-03-27T03:04:08.309 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:04:08.335 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:04:08.335 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:04:08.489 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:18 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:08.639 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:08.641 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-27T03:04:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-27T03:04:08.641 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-27T03:04:08.641 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:04:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-27T03:04:08.641 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-27T03:04:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-27T03:04:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-27T03:04:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-27T03:04:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-27T03:04:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-27T03:04:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-27T03:04:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-27T03:04:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-27T03:04:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-27T03:04:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-27T03:04:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-27T03:04:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-27T03:04:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:08.642 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:04:08.642 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-27T03:04:08.642 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-03-27T03:04:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:08.643 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.0 M 2024-03-27T03:04:08.643 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-03-27T03:04:08.643 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:04:08.696 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:04:08.696 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:04:09.124 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 150 kB/s | 18 kB 00:00 2024-03-27T03:04:09.149 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 167 kB/s | 24 kB 00:00 2024-03-27T03:04:09.208 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-27T03:04:09.233 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-27T03:04:09.258 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-27T03:04:09.283 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-27T03:04:09.309 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 171 kB/s | 52 kB 00:00 2024-03-27T03:04:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:04:09.401 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-27T03:04:09.426 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-27T03:04:09.476 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-27T03:04:09.507 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-27T03:04:09.535 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 897 kB/s | 203 kB 00:00 2024-03-27T03:04:09.589 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 362 kB/s | 59 kB 00:00 2024-03-27T03:04:09.612 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-27T03:04:09.619 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-27T03:04:09.660 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-27T03:04:09.777 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2024-03-27T03:04:09.779 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:09.779 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-27T03:04:09.779 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:04:09.792 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:04:09.792 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:04:09.838 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T03:04:09.867 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T03:04:09.884 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:04:09.885 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:04:09.953 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-27T03:04:10.401 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:04:10.556 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-27T03:04:10.595 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-27T03:04:10.612 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-27T03:04:10.667 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-27T03:04:10.704 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-27T03:04:10.714 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-27T03:04:10.745 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-27T03:04:10.847 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-27T03:04:10.948 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-27T03:04:11.005 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-27T03:04:11.724 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-27T03:04:11.777 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-27T03:04:11.811 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-27T03:04:11.859 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-27T03:04:11.881 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-27T03:04:11.963 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-27T03:04:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-27T03:04:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-27T03:04:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-27T03:04:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-27T03:04:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-27T03:04:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-27T03:04:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-27T03:04:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-27T03:04:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-27T03:04:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-27T03:04:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-27T03:04:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-27T03:04:12.628 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-27T03:04:12.989 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-27T03:04:12.991 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-27T03:04:12.992 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-27T03:04:12.992 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-27T03:04:12.992 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-27T03:04:12.992 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:12.992 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:13.009 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-27T03:04:13.246 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-03-27T03:04:13.806 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:23 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:13.956 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:13.957 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-27T03:04:13.957 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-27T03:04:13.957 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-27T03:04:13.957 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:04:13.958 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-27T03:04:13.958 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-27T03:04:13.958 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-27T03:04:13.958 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-27T03:04:13.958 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-27T03:04:13.958 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-27T03:04:13.958 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-27T03:04:13.958 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:13.958 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:04:13.958 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-27T03:04:13.958 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-03-27T03:04:13.958 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:13.959 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 6.8 M 2024-03-27T03:04:13.959 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 33 M 2024-03-27T03:04:13.959 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:04:14.404 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-03-27T03:04:14.487 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-27T03:04:14.621 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 309 kB/s | 138 kB 00:00 2024-03-27T03:04:14.646 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 364 kB/s | 58 kB 00:00 2024-03-27T03:04:14.771 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-27T03:04:14.796 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 291 kB/s | 51 kB 00:00 2024-03-27T03:04:14.798 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:14.798 INFO:teuthology.orchestra.run.smithi153.stdout:Total 8.1 MB/s | 6.8 MB 00:00 2024-03-27T03:04:14.810 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:04:14.820 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:04:14.820 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:04:14.908 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:04:14.909 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:04:15.204 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:04:15.369 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-27T03:04:15.423 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-27T03:04:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-27T03:04:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-27T03:04:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-27T03:04:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-27T03:04:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-27T03:04:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T03:04:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T03:04:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-27T03:04:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-27T03:04:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-27T03:04:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-27T03:04:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-27T03:04:15.939 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-27T03:04:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-27T03:04:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:16.008 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:04:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:04:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-27T03:04:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-27T03:04:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-27T03:04:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-27T03:04:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-27T03:04:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-27T03:04:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-27T03:04:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-27T03:04:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-27T03:04:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-27T03:04:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-27T03:04:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:16.009 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:16.024 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-27T03:04:16.086 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-27T03:04:16.112 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-27T03:04:16.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-rook 2024-03-27T03:04:16.928 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-27T03:04:16.928 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-27T03:04:16.928 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-27T03:04:16.929 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-27T03:04:16.929 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-27T03:04:16.929 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-27T03:04:16.935 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:19 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:17.084 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:17.085 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-27T03:04:17.085 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-27T03:04:17.085 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-27T03:04:17.085 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:04:17.085 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-27T03:04:17.085 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-27T03:04:17.085 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-27T03:04:17.085 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-27T03:04:17.085 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-27T03:04:17.085 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-27T03:04:17.085 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-27T03:04:17.086 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-27T03:04:17.086 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-27T03:04:17.086 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-27T03:04:17.086 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-27T03:04:17.086 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-27T03:04:17.086 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-27T03:04:17.086 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-27T03:04:17.086 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:17.086 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:04:17.086 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-27T03:04:17.086 INFO:teuthology.orchestra.run.smithi196.stdout:Install 13 Packages 2024-03-27T03:04:17.086 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:17.087 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.0 M 2024-03-27T03:04:17.087 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-03-27T03:04:17.087 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:04:17.221 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-27T03:04:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:17.222 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:04:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:04:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-27T03:04:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-27T03:04:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-27T03:04:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-27T03:04:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-27T03:04:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:17.222 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:17.462 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-03-27T03:04:17.518 INFO:teuthology.orchestra.run.smithi196.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 131 kB/s | 18 kB 00:00 2024-03-27T03:04:17.535 INFO:teuthology.orchestra.run.smithi196.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 157 kB/s | 24 kB 00:00 2024-03-27T03:04:17.593 INFO:teuthology.orchestra.run.smithi196.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-27T03:04:17.619 INFO:teuthology.orchestra.run.smithi196.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-27T03:04:17.644 INFO:teuthology.orchestra.run.smithi196.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-27T03:04:17.669 INFO:teuthology.orchestra.run.smithi196.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-27T03:04:17.695 INFO:teuthology.orchestra.run.smithi196.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 166 kB/s | 52 kB 00:00 2024-03-27T03:04:17.920 INFO:teuthology.orchestra.run.smithi196.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-27T03:04:17.945 INFO:teuthology.orchestra.run.smithi196.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-03-27T03:04:18.025 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:28 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:18.171 INFO:teuthology.orchestra.run.smithi196.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 263 kB/s | 59 kB 00:00 2024-03-27T03:04:18.175 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:18.175 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:18.175 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-27T03:04:18.175 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:18.176 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:04:18.176 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-27T03:04:18.176 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-27T03:04:18.176 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-27T03:04:18.176 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:18.176 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:04:18.176 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:18.176 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-03-27T03:04:18.176 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:18.177 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 964 k 2024-03-27T03:04:18.177 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.8 M 2024-03-27T03:04:18.177 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:04:18.204 INFO:teuthology.orchestra.run.smithi196.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 398 kB/s | 203 kB 00:00 2024-03-27T03:04:18.308 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 602 kB/s | 79 kB 00:00 2024-03-27T03:04:18.346 INFO:teuthology.orchestra.run.smithi196.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-03-27T03:04:18.471 INFO:teuthology.orchestra.run.smithi196.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-27T03:04:18.473 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:18.473 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-27T03:04:18.474 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:04:18.489 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:04:18.489 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:04:18.584 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:04:18.586 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:04:18.883 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 885 kB 00:00 2024-03-27T03:04:18.884 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:18.885 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.3 MB/s | 964 kB 00:00 2024-03-27T03:04:18.885 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:04:18.906 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:04:18.906 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:04:18.980 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:04:18.980 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:04:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:04:19.168 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:04:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-27T03:04:19.309 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-27T03:04:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-27T03:04:19.377 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-27T03:04:19.383 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-27T03:04:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-27T03:04:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-27T03:04:19.557 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-27T03:04:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-27T03:04:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-27T03:04:20.091 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-27T03:04:20.091 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-27T03:04:20.398 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-27T03:04:20.399 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:20.399 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:04:20.399 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-27T03:04:20.399 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:20.399 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:20.443 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-27T03:04:20.488 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-27T03:04:20.521 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-27T03:04:20.570 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-27T03:04:20.592 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-27T03:04:20.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-volume 2024-03-27T03:04:21.247 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:31 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:21.398 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:21.398 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:21.398 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-27T03:04:21.398 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:21.398 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:04:21.398 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-27T03:04:21.399 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-27T03:04:21.399 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-27T03:04:21.399 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:21.399 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:04:21.399 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:21.399 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-03-27T03:04:21.399 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:21.399 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 332 k 2024-03-27T03:04:21.399 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.5 M 2024-03-27T03:04:21.400 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:04:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-27T03:04:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-27T03:04:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-27T03:04:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-27T03:04:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-27T03:04:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-27T03:04:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-27T03:04:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-27T03:04:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-27T03:04:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-27T03:04:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-27T03:04:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-27T03:04:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-27T03:04:21.537 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 501 kB/s | 69 kB 00:00 2024-03-27T03:04:21.679 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 945 kB/s | 264 kB 00:00 2024-03-27T03:04:21.680 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:21.680 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-27T03:04:21.681 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:04:21.689 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:04:21.689 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:04:21.729 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:04:21.729 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:04:21.787 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-27T03:04:21.787 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:21.787 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:04:21.787 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:04:21.787 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-27T03:04:21.787 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-27T03:04:21.788 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-27T03:04:21.788 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-27T03:04:21.788 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-27T03:04:21.788 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-27T03:04:21.788 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-27T03:04:21.788 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-27T03:04:21.788 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-27T03:04:21.788 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-27T03:04:21.788 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-27T03:04:21.788 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-27T03:04:21.788 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:21.788 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:21.878 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:04:22.018 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-cephadm 2024-03-27T03:04:22.072 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-27T03:04:22.106 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-27T03:04:22.122 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-27T03:04:22.122 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:04:22.123 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-27T03:04:22.123 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:22.582 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:25 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:22.733 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:22.734 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-27T03:04:22.734 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-27T03:04:22.734 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-27T03:04:22.734 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:04:22.734 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-27T03:04:22.734 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-27T03:04:22.734 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-27T03:04:22.734 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-27T03:04:22.735 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-27T03:04:22.735 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-27T03:04:22.735 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-27T03:04:22.735 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:22.735 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:04:22.735 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-27T03:04:22.735 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-03-27T03:04:22.735 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:22.736 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 6.8 M 2024-03-27T03:04:22.736 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 33 M 2024-03-27T03:04:22.736 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:04:22.899 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-27T03:04:23.233 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-27T03:04:23.233 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:23.233 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:04:23.233 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:04:23.233 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-27T03:04:23.233 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:23.233 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:23.550 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-03-27T03:04:23.623 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-27T03:04:23.740 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 352 kB/s | 138 kB 00:00 2024-03-27T03:04:23.899 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-03-27T03:04:23.924 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 317 kB/s | 58 kB 00:00 2024-03-27T03:04:23.949 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-27T03:04:24.132 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:34 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:24.281 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:24.281 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-27T03:04:24.282 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:24.282 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:04:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-27T03:04:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:24.282 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:04:24.282 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:24.282 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-27T03:04:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:24.282 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 127 k 2024-03-27T03:04:24.282 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 456 k 2024-03-27T03:04:24.282 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:04:24.366 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 244 kB/s | 248 kB 00:01 2024-03-27T03:04:24.369 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:24.369 INFO:teuthology.orchestra.run.smithi196.stdout:Total 4.2 MB/s | 6.8 MB 00:01 2024-03-27T03:04:24.382 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:04:24.394 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:04:24.394 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:04:24.481 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:04:24.482 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:04:24.668 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 329 kB/s | 127 kB 00:00 2024-03-27T03:04:24.668 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:24.669 INFO:teuthology.orchestra.run.smithi153.stdout:Total 328 kB/s | 127 kB 00:00 2024-03-27T03:04:24.669 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:04:24.674 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:04:24.674 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:04:24.732 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:04:24.733 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:04:24.782 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:04:24.875 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:04:24.964 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-27T03:04:24.983 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:25.011 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-27T03:04:25.472 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:25.519 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-27T03:04:25.605 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-27T03:04:25.666 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-27T03:04:25.690 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-27T03:04:25.743 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:25.743 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:25.743 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:04:25.743 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:04:25.743 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:25.743 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:25.976 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-03-27T03:04:26.476 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-27T03:04:26.476 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-27T03:04:26.476 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-27T03:04:26.477 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-27T03:04:26.477 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-27T03:04:26.477 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-27T03:04:26.547 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:36 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:26.648 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:26.703 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:26.704 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-27T03:04:26.704 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:26.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-03-27T03:04:26.786 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-27T03:04:26.786 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:26.786 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:04:26.786 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:04:26.786 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-27T03:04:26.786 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-27T03:04:26.786 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-27T03:04:26.786 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-27T03:04:26.786 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-27T03:04:26.786 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:26.787 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:27.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse 2024-03-27T03:04:27.331 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:37 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:27.481 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:27.481 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:27.481 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-27T03:04:27.481 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:27.482 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:04:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-27T03:04:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:27.482 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:04:27.482 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:27.482 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-27T03:04:27.482 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:27.482 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 31 k 2024-03-27T03:04:27.482 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 140 k 2024-03-27T03:04:27.482 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:04:27.583 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:30 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:27.653 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 182 kB/s | 31 kB 00:00 2024-03-27T03:04:27.654 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:27.654 INFO:teuthology.orchestra.run.smithi153.stdout:Total 181 kB/s | 31 kB 00:00 2024-03-27T03:04:27.654 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:04:27.658 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:04:27.658 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:04:27.686 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:04:27.686 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:04:27.733 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:27.734 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:27.734 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-27T03:04:27.734 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:27.734 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:04:27.734 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-27T03:04:27.734 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-27T03:04:27.734 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-27T03:04:27.734 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:27.734 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:04:27.734 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:27.734 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-03-27T03:04:27.735 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:27.735 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 964 k 2024-03-27T03:04:27.735 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.8 M 2024-03-27T03:04:27.735 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:04:27.805 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:04:27.886 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 521 kB/s | 79 kB 00:00 2024-03-27T03:04:27.893 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:28.268 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:28.536 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 885 kB 00:00 2024-03-27T03:04:28.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:28.537 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:28.537 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:04:28.537 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:04:28.537 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:28.537 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:28.538 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:28.538 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.2 MB/s | 964 kB 00:00 2024-03-27T03:04:28.538 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:04:28.559 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:04:28.559 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:04:28.633 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:04:28.633 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:04:28.702 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-03-27T03:04:28.819 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:04:28.972 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-27T03:04:29.045 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-27T03:04:29.273 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:39 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:29.369 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:29.421 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:29.422 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-27T03:04:29.422 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:29.483 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-03-27T03:04:29.701 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-27T03:04:29.701 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-27T03:04:29.998 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-27T03:04:29.998 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:29.998 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:04:29.998 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-27T03:04:29.998 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:29.999 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:30.047 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:40 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:30.144 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:30.197 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:30.198 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-27T03:04:30.198 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:30.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-03-27T03:04:30.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-volume 2024-03-27T03:04:30.813 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:40 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:30.850 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:33 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:30.908 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:30.960 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:30.961 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-27T03:04:30.961 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:31.006 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:31.006 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:31.006 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-27T03:04:31.006 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:31.006 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:04:31.006 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-27T03:04:31.007 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-27T03:04:31.007 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-27T03:04:31.007 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:31.007 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:04:31.007 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:31.007 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-03-27T03:04:31.007 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:31.008 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 332 k 2024-03-27T03:04:31.008 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 1.5 M 2024-03-27T03:04:31.008 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:04:31.009 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-03-27T03:04:31.149 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 486 kB/s | 69 kB 00:00 2024-03-27T03:04:31.450 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 597 kB/s | 264 kB 00:00 2024-03-27T03:04:31.451 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:31.451 INFO:teuthology.orchestra.run.smithi196.stdout:Total 748 kB/s | 332 kB 00:00 2024-03-27T03:04:31.451 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:04:31.460 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:04:31.460 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:04:31.497 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:04:31.497 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:04:31.561 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:41 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:31.646 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:04:31.658 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:31.710 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:31.711 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-27T03:04:31.711 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:31.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-03-27T03:04:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-27T03:04:31.860 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-27T03:04:31.876 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-27T03:04:31.876 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:04:31.876 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-27T03:04:31.876 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:32.318 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:42 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:32.416 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:32.467 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:32.469 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-27T03:04:32.469 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:32.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-03-27T03:04:32.653 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-27T03:04:32.977 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-27T03:04:32.977 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:32.977 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:04:32.977 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-27T03:04:32.977 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-27T03:04:32.977 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:32.978 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:33.075 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:43 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:33.172 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:33.223 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:33.224 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-27T03:04:33.224 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:33.286 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-03-27T03:04:33.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel 2024-03-27T03:04:33.846 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:43 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:33.877 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:36 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:33.994 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:33.995 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:33.995 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-27T03:04:33.995 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:33.995 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:04:33.995 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-27T03:04:33.995 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:33.995 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:04:33.995 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:33.995 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-27T03:04:33.995 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:33.995 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 85 k 2024-03-27T03:04:33.995 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 231 k 2024-03-27T03:04:33.996 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:04:34.026 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:34.027 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:34.027 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-27T03:04:34.027 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:34.027 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:04:34.027 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-27T03:04:34.027 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:34.027 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:04:34.027 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:34.028 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-27T03:04:34.028 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:34.028 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 127 k 2024-03-27T03:04:34.028 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 456 k 2024-03-27T03:04:34.028 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:04:34.247 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 579 kB/s | 127 kB 00:00 2024-03-27T03:04:34.248 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:34.248 INFO:teuthology.orchestra.run.smithi196.stdout:Total 575 kB/s | 127 kB 00:00 2024-03-27T03:04:34.248 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:04:34.254 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:04:34.254 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:04:34.310 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:04:34.310 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:04:34.314 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 267 kB/s | 85 kB 00:00 2024-03-27T03:04:34.315 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:34.315 INFO:teuthology.orchestra.run.smithi153.stdout:Total 266 kB/s | 85 kB 00:00 2024-03-27T03:04:34.316 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:04:34.322 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:04:34.323 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:04:34.378 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:04:34.378 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:04:34.451 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:04:34.520 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:04:34.563 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:34.613 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:35.019 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:35.037 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:35.296 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:35.296 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:35.296 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:04:35.296 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:04:35.296 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:35.296 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:35.337 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:04:35.337 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:04:35.338 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:35.338 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:35.516 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2 2024-03-27T03:04:35.574 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-03-27T03:04:36.088 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:39 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:36.166 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:46 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:36.192 INFO:teuthology.orchestra.run.smithi196.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:36.248 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:36.249 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-27T03:04:36.249 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:36.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs-devel 2024-03-27T03:04:36.322 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:36.322 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:36.322 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-27T03:04:36.322 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:36.322 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:04:36.322 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-27T03:04:36.323 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:36.323 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:04:36.323 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:36.323 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-27T03:04:36.323 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:36.323 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.1 M 2024-03-27T03:04:36.323 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 13 M 2024-03-27T03:04:36.324 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:04:36.871 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:39 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:37.020 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:37.021 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-27T03:04:37.021 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:37.021 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:04:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-27T03:04:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:37.021 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:04:37.022 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:37.022 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-27T03:04:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:37.022 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 31 k 2024-03-27T03:04:37.022 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 140 k 2024-03-27T03:04:37.022 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:04:37.134 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 3.8 MB/s | 3.1 MB 00:00 2024-03-27T03:04:37.135 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:37.135 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.8 MB/s | 3.1 MB 00:00 2024-03-27T03:04:37.135 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:04:37.144 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:04:37.144 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:04:37.201 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 174 kB/s | 31 kB 00:00 2024-03-27T03:04:37.201 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:37.202 INFO:teuthology.orchestra.run.smithi196.stdout:Total 173 kB/s | 31 kB 00:00 2024-03-27T03:04:37.202 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:04:37.205 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:04:37.205 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:04:37.219 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:04:37.219 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:04:37.244 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:04:37.244 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:04:37.352 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:04:37.465 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:37.653 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:04:37.718 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:37.743 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:37.744 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:04:37.744 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-27T03:04:37.744 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T03:04:37.744 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T03:04:37.744 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:37.856 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:38.122 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:04:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:04:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:38.122 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:38.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2 2024-03-27T03:04:38.891 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:41 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:38.897 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:38.897 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:38.897 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:04:38.897 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:04:38.897 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:38.897 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:38.987 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:39.039 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:39.040 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-27T03:04:39.040 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:39.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1 2024-03-27T03:04:39.119 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-03-27T03:04:39.653 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:42 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:39.696 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:49 ago on Wed 27 Mar 2024 03:02:50 AM UTC. 2024-03-27T03:04:39.750 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:39.805 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:39.806 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-27T03:04:39.806 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:39.845 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-27T03:04:39.846 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:39.846 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-27T03:04:39.846 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:39.846 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-27T03:04:39.847 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-27T03:04:39.847 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:39.847 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-27T03:04:39.847 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-27T03:04:39.847 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-27T03:04:39.847 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:39.847 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 171 k 2024-03-27T03:04:39.847 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 494 k 2024-03-27T03:04:39.847 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-27T03:04:39.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados 2024-03-27T03:04:40.195 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 493 kB/s | 171 kB 00:00 2024-03-27T03:04:40.196 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:40.196 INFO:teuthology.orchestra.run.smithi153.stdout:Total 490 kB/s | 171 kB 00:00 2024-03-27T03:04:40.196 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-27T03:04:40.204 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-27T03:04:40.204 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-27T03:04:40.262 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-27T03:04:40.262 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-27T03:04:40.411 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:43 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:40.416 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-27T03:04:40.508 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:40.520 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:40.561 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:40.562 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-27T03:04:40.562 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:40.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw 2024-03-27T03:04:40.943 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:41.175 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:44 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:41.219 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:41.219 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:41.219 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-27T03:04:41.219 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:04:41.219 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:41.219 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-27T03:04:41.272 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:41.326 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:41.327 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-27T03:04:41.327 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:41.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs 2024-03-27T03:04:41.443 DEBUG:teuthology.parallel:result is None 2024-03-27T03:04:41.939 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:44 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:42.036 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:42.087 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:42.088 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-27T03:04:42.088 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:42.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd 2024-03-27T03:04:42.702 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:45 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:42.798 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-27T03:04:42.849 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:42.850 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-27T03:04:42.850 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:42.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse 2024-03-27T03:04:43.470 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:46 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:43.628 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 85 k 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 231 k 2024-03-27T03:04:43.629 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:04:43.846 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 394 kB/s | 85 kB 00:00 2024-03-27T03:04:43.847 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:43.847 INFO:teuthology.orchestra.run.smithi196.stdout:Total 391 kB/s | 85 kB 00:00 2024-03-27T03:04:43.847 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:04:43.854 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:04:43.855 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:04:43.910 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:04:43.911 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:04:44.048 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:04:44.146 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:44.530 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:44.821 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:44.821 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:44.821 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:04:44.822 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:04:44.822 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:44.822 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:45.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2024-03-27T03:04:45.653 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:48 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:45.808 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:45.809 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:45.809 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-27T03:04:45.809 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:45.809 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:04:45.809 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-27T03:04:45.810 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:45.810 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:04:45.810 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:45.810 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-27T03:04:45.810 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:45.810 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.1 M 2024-03-27T03:04:45.810 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 13 M 2024-03-27T03:04:45.810 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:04:46.433 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.0 MB/s | 3.1 MB 00:00 2024-03-27T03:04:46.433 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:46.433 INFO:teuthology.orchestra.run.smithi196.stdout:Total 4.9 MB/s | 3.1 MB 00:00 2024-03-27T03:04:46.434 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:04:46.443 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:04:46.443 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:04:46.517 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:04:46.517 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:04:46.947 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:04:47.000 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:47.025 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:47.025 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T03:04:47.026 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-27T03:04:47.026 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T03:04:47.026 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T03:04:47.026 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:48.229 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:04:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:04:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:48.229 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:48.497 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd 2024-03-27T03:04:49.073 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:52 ago on Wed 27 Mar 2024 03:02:57 AM UTC. 2024-03-27T03:04:49.224 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 171 k 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 494 k 2024-03-27T03:04:49.225 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-27T03:04:49.562 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 511 kB/s | 171 kB 00:00 2024-03-27T03:04:49.563 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-27T03:04:49.563 INFO:teuthology.orchestra.run.smithi196.stdout:Total 508 kB/s | 171 kB 00:00 2024-03-27T03:04:49.563 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-27T03:04:49.570 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-27T03:04:49.571 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-27T03:04:49.626 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-27T03:04:49.627 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-27T03:04:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-27T03:04:49.876 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:50.342 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:50.639 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-27T03:04:50.639 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:50.639 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-27T03:04:50.639 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-27T03:04:50.639 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:50.640 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-27T03:04:50.907 DEBUG:teuthology.parallel:result is None 2024-03-27T03:04:50.908 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-27T03:04:51.136 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-27T03:04:51.161 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-27T03:04:51.162 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-27T03:04:51.162 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-27T03:04:51.164 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-27T03:04:51.306 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-27T03:04:51.327 INFO:teuthology.orchestra.run.smithi196.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-27T03:04:51.328 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-27T03:04:51.328 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-27T03:04:51.330 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-27T03:04:51.330 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:04:51.330 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-27T03:04:51.364 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:04:51.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-27T03:04:51.395 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-27T03:04:51.396 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:04:51.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-03-27T03:04:51.433 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-27T03:04:51.502 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:04:51.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-03-27T03:04:51.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-27T03:04:51.596 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-27T03:04:51.597 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:04:51.597 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-27T03:04:51.625 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-27T03:04:51.694 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:04:51.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-27T03:04:51.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-27T03:04:51.789 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-27T03:04:51.790 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:04:51.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-03-27T03:04:51.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-27T03:04:51.885 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:04:51.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-03-27T03:04:51.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-27T03:04:51.980 INFO:teuthology.run_tasks:Running task ceph... 2024-03-27T03:04:52.071 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-27T03:04:52.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2024-03-27T03:04:52.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /var/log/ceph 2024-03-27T03:04:52.101 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-27T03:04:52.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-27T03:04:52.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-27T03:04:52.168 INFO:tasks.ceph:Creating extra log directories... 2024-03-27T03:04:52.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-27T03:04:52.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-27T03:04:52.246 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-27T03:04:52.247 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-27T03:04:52.247 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622698', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCE2dRf1adqYcZW5xkSEOuI5DfuEsx1HXm8zZPbhMJqUbT0jV1f7DJMaWOlAr3wqgK+GdDh9ylXyGJ1Xn1vnopk=', 'smithi196.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFZ5kbULDe0hPvOIpHU+FY7udHbBIdr9lbE57zrSjNk6OqDJDxHa8jlUaa1AYKifJLnaUM/ut1vO0rQodJQVeaw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-27T03:04:52.247 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-27T03:04:52.278 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-27T03:04:52.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2024-03-27T03:04:52.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/run/ceph 2024-03-27T03:04:52.360 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:04:52.360 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-03-27T03:04:52.413 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-27T03:04:52.413 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-03-27T03:04:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-27T03:04:52.468 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T03:04:52.469 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-27T03:04:52.469 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T03:04:52.469 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T03:04:52.469 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-27 03:04:38.000704804 +0000 2024-03-27T03:04:52.469 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-27 03:01:50.968270728 +0000 2024-03-27T03:04:52.469 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-27 03:01:50.968270728 +0000 2024-03-27T03:04:52.469 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-27 03:01:50.968270728 +0000 2024-03-27T03:04:52.469 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-27T03:04:52.534 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-27T03:04:52.534 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-27T03:04:52.534 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000287908 s, 1.8 MB/s 2024-03-27T03:04:52.535 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-27T03:04:52.591 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-03-27T03:04:52.646 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-27T03:04:52.646 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T03:04:52.646 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-27T03:04:52.646 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T03:04:52.646 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T03:04:52.647 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-27 03:04:38.000704804 +0000 2024-03-27T03:04:52.647 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-27 03:01:51.492260754 +0000 2024-03-27T03:04:52.647 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-27 03:01:51.492260754 +0000 2024-03-27T03:04:52.647 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-27 03:01:51.492260754 +0000 2024-03-27T03:04:52.647 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-27T03:04:52.712 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-27T03:04:52.712 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-27T03:04:52.713 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000184553 s, 2.8 MB/s 2024-03-27T03:04:52.714 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-27T03:04:52.769 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-03-27T03:04:52.828 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-27T03:04:52.828 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T03:04:52.828 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-27T03:04:52.828 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T03:04:52.828 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T03:04:52.829 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-27 03:04:38.000704804 +0000 2024-03-27T03:04:52.829 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-27 03:01:51.824254434 +0000 2024-03-27T03:04:52.829 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-27 03:01:51.824254434 +0000 2024-03-27T03:04:52.829 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-27 03:01:51.824254434 +0000 2024-03-27T03:04:52.829 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-27T03:04:52.893 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-27T03:04:52.893 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-27T03:04:52.893 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000293828 s, 1.7 MB/s 2024-03-27T03:04:52.894 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-27T03:04:52.950 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-03-27T03:04:53.005 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-27T03:04:53.005 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T03:04:53.006 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-27T03:04:53.006 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T03:04:53.006 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T03:04:53.006 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-27 03:04:38.001704785 +0000 2024-03-27T03:04:53.006 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-27 03:01:52.160248038 +0000 2024-03-27T03:04:53.006 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-27 03:01:52.160248038 +0000 2024-03-27T03:04:53.006 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-27 03:01:52.160248038 +0000 2024-03-27T03:04:53.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-27T03:04:53.071 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-27T03:04:53.071 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-27T03:04:53.071 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.00021995 s, 2.3 MB/s 2024-03-27T03:04:53.072 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-27T03:04:53.128 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T03:04:53.128 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:04:53.128 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-03-27T03:04:53.147 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-27T03:04:53.147 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-03-27T03:04:53.204 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-27T03:04:53.204 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T03:04:53.204 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-27T03:04:53.204 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T03:04:53.204 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T03:04:53.204 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-27 03:04:47.307869310 +0000 2024-03-27T03:04:53.204 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-27 03:01:50.970180305 +0000 2024-03-27T03:04:53.204 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-27 03:01:50.970180305 +0000 2024-03-27T03:04:53.204 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-27 03:01:50.970180305 +0000 2024-03-27T03:04:53.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-27T03:04:53.273 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-27T03:04:53.273 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-27T03:04:53.273 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000270409 s, 1.9 MB/s 2024-03-27T03:04:53.274 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-27T03:04:53.332 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-03-27T03:04:53.389 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-27T03:04:53.389 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T03:04:53.390 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-27T03:04:53.390 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T03:04:53.390 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T03:04:53.390 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-27 03:04:47.307869310 +0000 2024-03-27T03:04:53.390 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-27 03:01:51.325173032 +0000 2024-03-27T03:04:53.390 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-27 03:01:51.325173032 +0000 2024-03-27T03:04:53.390 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-27 03:01:51.325173032 +0000 2024-03-27T03:04:53.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-27T03:04:53.458 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-27T03:04:53.459 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-27T03:04:53.459 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000223878 s, 2.3 MB/s 2024-03-27T03:04:53.460 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-27T03:04:53.517 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-03-27T03:04:53.573 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-27T03:04:53.573 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T03:04:53.573 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-27T03:04:53.573 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T03:04:53.573 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T03:04:53.573 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-27 03:04:47.307869310 +0000 2024-03-27T03:04:53.573 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-27 03:01:51.679165781 +0000 2024-03-27T03:04:53.573 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-27 03:01:51.679165781 +0000 2024-03-27T03:04:53.573 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-27 03:01:51.679165781 +0000 2024-03-27T03:04:53.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-27T03:04:53.640 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-27T03:04:53.640 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-27T03:04:53.640 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.00019193 s, 2.7 MB/s 2024-03-27T03:04:53.641 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-27T03:04:53.698 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-03-27T03:04:53.754 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-27T03:04:53.754 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T03:04:53.754 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-27T03:04:53.755 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T03:04:53.755 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T03:04:53.755 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-27 03:04:47.308869289 +0000 2024-03-27T03:04:53.755 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-27 03:01:52.011158979 +0000 2024-03-27T03:04:53.755 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-27 03:01:52.011158979 +0000 2024-03-27T03:04:53.755 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-27 03:01:52.011158979 +0000 2024-03-27T03:04:53.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-27T03:04:53.821 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-27T03:04:53.822 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-27T03:04:53.822 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000219254 s, 2.3 MB/s 2024-03-27T03:04:53.823 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-27T03:04:53.879 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T03:04:53.879 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-27T03:04:53.879 INFO:tasks.ceph:Generating config... 2024-03-27T03:04:53.880 INFO:tasks.ceph:[client] debug monc = 20 2024-03-27T03:04:53.881 INFO:tasks.ceph:[client] debug ms = 1 2024-03-27T03:04:53.881 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-27T03:04:53.881 INFO:tasks.ceph:[client] debug rados = 20 2024-03-27T03:04:53.881 INFO:tasks.ceph:[global] lockdep = True 2024-03-27T03:04:53.881 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-27T03:04:53.881 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-27T03:04:53.881 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-27T03:04:53.881 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-27T03:04:53.881 INFO:tasks.ceph:[global] ms type = async 2024-03-27T03:04:53.881 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-27T03:04:53.881 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-27T03:04:53.881 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-27T03:04:53.881 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-27T03:04:53.881 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-27T03:04:53.881 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-27T03:04:53.882 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-27T03:04:53.882 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-27T03:04:53.882 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-27T03:04:53.882 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-27T03:04:53.882 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-27T03:04:53.882 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-27T03:04:53.882 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-27T03:04:53.882 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-27T03:04:53.882 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-27T03:04:53.882 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-27T03:04:53.882 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-27T03:04:53.883 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-27T03:04:53.884 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-27T03:04:53.884 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-27T03:04:53.884 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-27T03:04:53.884 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-27T03:04:53.884 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-27T03:04:53.884 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-27T03:04:53.884 INFO:tasks.ceph:Setting up mon.a... 2024-03-27T03:04:53.885 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-27T03:04:53.942 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.keyring 2024-03-27T03:04:53.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-27T03:04:53.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T03:04:54.017 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.153:6789]'), ('mon.c', '[v1:172.21.15.153:6790]'), ('mon.b', '[v1:172.21.15.196:6789]')] 2024-03-27T03:04:54.018 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.153:6789],[v1:172.21.15.153:6790],[v1:172.21.15.196:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-27T03:04:54.019 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:04:54.019 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-27T03:04:54.073 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.153:6789]' --addv c '[v1:172.21.15.153:6790]' --addv b '[v1:172.21.15.196:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:04:54.157 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:04:54.158 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: generated fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:04:54.158 INFO:teuthology.orchestra.run.smithi153.stdout:setting min_mon_release = quincy 2024-03-27T03:04:54.158 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 0 2024-03-27T03:04:54.158 INFO:teuthology.orchestra.run.smithi153.stdout:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:04:54.158 INFO:teuthology.orchestra.run.smithi153.stdout:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:04:54.158 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:04:54.158 INFO:teuthology.orchestra.run.smithi153.stdout:min_mon_release 17 (quincy) 2024-03-27T03:04:54.158 INFO:teuthology.orchestra.run.smithi153.stdout:election_strategy: 1 2024-03-27T03:04:54.158 INFO:teuthology.orchestra.run.smithi153.stdout:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:04:54.158 INFO:teuthology.orchestra.run.smithi153.stdout:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:04:54.158 INFO:teuthology.orchestra.run.smithi153.stdout:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:04:54.158 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-27T03:04:54.159 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-27T03:04:54.174 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 318002cf-3baf-46ae-b53f-8860af9c7a7a... 2024-03-27T03:04:54.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-27T03:04:54.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-27T03:04:54.265 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-03-27T03:04:54.265 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2024-03-27T03:04:54.265 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-27T03:04:54.265 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2024-03-27T03:04:54.265 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.265 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2024-03-27T03:04:54.265 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.265 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2024-03-27T03:04:54.265 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.265 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2024-03-27T03:04:54.265 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2024-03-27T03:04:54.265 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2024-03-27T03:04:54.266 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2024-03-27T03:04:54.267 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = "[v1:172.21.15.153:6789],[v1:172.21.15.153:6790],[v1:172.21.15.196:6789]" 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: lockdep = True 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: mon client directed command retry = 5 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: mon election default strategy = 1 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: ms bind msgr2 = False 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject socket failures = 5000 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: ms type = async 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2024-03-27T03:04:54.268 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2024-03-27T03:04:54.269 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore compression algorithm = snappy 2024-03-27T03:04:54.270 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore compression mode = aggressive 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore zero block detection = True 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 20 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 20 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 10 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: osd class default list = * 2024-03-27T03:04:54.271 INFO:teuthology.orchestra.run.smithi153.stdout: osd class load list = * 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: osd client watch timeout = 120 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify cached snaps = True 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify missing on start = True 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock override recovery settings = True 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue = debug_random 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue cut off = debug_random 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: osd shutdown pgref assert = True 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-03-27T03:04:54.272 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-03-27T03:04:54.273 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-03-27T03:04:54.273 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-03-27T03:04:54.273 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2024-03-27T03:04:54.273 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.273 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2024-03-27T03:04:54.273 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-27T03:04:54.273 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-03-27T03:04:54.273 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2024-03-27T03:04:54.273 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-03-27T03:04:54.273 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2024-03-27T03:04:54.273 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2024-03-27T03:04:54.273 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2024-03-27T03:04:54.275 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2024-03-27T03:04:54.275 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.275 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2024-03-27T03:04:54.275 INFO:teuthology.orchestra.run.smithi153.stdout: mon min osdmap epochs = 50 2024-03-27T03:04:54.275 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune interval = 2 2024-03-27T03:04:54.275 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune min = 15 2024-03-27T03:04:54.275 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune txsize = 2 2024-03-27T03:04:54.275 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub interval = 300 2024-03-27T03:04:54.275 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on pool no app = False 2024-03-27T03:04:54.275 INFO:teuthology.orchestra.run.smithi153.stdout: paxos service trim min = 10 2024-03-27T03:04:54.275 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:04:54.275 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2024-03-27T03:04:54.276 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2024-03-27T03:04:54.276 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2024-03-27T03:04:54.276 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2024-03-27T03:04:54.276 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-27T03:04:54.276 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-27T03:04:54.276 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2024-03-27T03:04:54.276 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-27T03:04:54.276 INFO:teuthology.orchestra.run.smithi153.stdout: debug objecter = 20 2024-03-27T03:04:54.276 INFO:teuthology.orchestra.run.smithi153.stdout: debug rados = 20 2024-03-27T03:04:54.276 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2024-03-27T03:04:54.276 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.c] 2024-03-27T03:04:54.276 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.b] 2024-03-27T03:04:54.277 INFO:teuthology.orchestra.run.smithi196.stdout:[global] 2024-03-27T03:04:54.277 INFO:teuthology.orchestra.run.smithi196.stdout: chdir = "" 2024-03-27T03:04:54.277 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-27T03:04:54.277 INFO:teuthology.orchestra.run.smithi196.stdout: auth supported = cephx 2024-03-27T03:04:54.277 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.277 INFO:teuthology.orchestra.run.smithi196.stdout: filestore xattr use omap = true 2024-03-27T03:04:54.277 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.278 INFO:teuthology.orchestra.run.smithi196.stdout: mon clock drift allowed = 1.000 2024-03-27T03:04:54.278 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.278 INFO:teuthology.orchestra.run.smithi196.stdout: osd crush chooseleaf type = 0 2024-03-27T03:04:54.278 INFO:teuthology.orchestra.run.smithi196.stdout: auth debug = true 2024-03-27T03:04:54.278 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.278 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on old message = true 2024-03-27T03:04:54.278 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on bug = true 2024-03-27T03:04:54.278 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.278 INFO:teuthology.orchestra.run.smithi196.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-27T03:04:54.278 INFO:teuthology.orchestra.run.smithi196.stdout: mon pg warn max object skew = 0 2024-03-27T03:04:54.278 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.278 INFO:teuthology.orchestra.run.smithi196.stdout: # disable pg_autoscaler by default for new pools 2024-03-27T03:04:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-27T03:04:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default size = 2 2024-03-27T03:04:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow primary affinity = true 2024-03-27T03:04:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow pg remap = true 2024-03-27T03:04:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on legacy crush tunables = false 2024-03-27T03:04:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on crush straw calc version zero = false 2024-03-27T03:04:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on no sortbitwise = false 2024-03-27T03:04:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on osd down out interval zero = false 2024-03-27T03:04:54.279 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on too few osds = false 2024-03-27T03:04:54.280 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-27T03:04:54.280 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-27T03:04:54.280 INFO:teuthology.orchestra.run.smithi196.stdout: mon_allow_pool_size_one = true 2024-03-27T03:04:54.280 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.280 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-27T03:04:54.280 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.280 INFO:teuthology.orchestra.run.smithi196.stdout: osd default data pool replay window = 5 2024-03-27T03:04:54.280 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.280 INFO:teuthology.orchestra.run.smithi196.stdout: mon allow pool delete = true 2024-03-27T03:04:54.280 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.280 INFO:teuthology.orchestra.run.smithi196.stdout: mon cluster log file level = debug 2024-03-27T03:04:54.281 INFO:teuthology.orchestra.run.smithi196.stdout: debug asserts on shutdown = true 2024-03-27T03:04:54.281 INFO:teuthology.orchestra.run.smithi196.stdout: mon health detail to clog = false 2024-03-27T03:04:54.281 INFO:teuthology.orchestra.run.smithi196.stdout: mon host = "[v1:172.21.15.153:6789],[v1:172.21.15.153:6790],[v1:172.21.15.196:6789]" 2024-03-27T03:04:54.281 INFO:teuthology.orchestra.run.smithi196.stdout: lockdep = True 2024-03-27T03:04:54.281 INFO:teuthology.orchestra.run.smithi196.stdout: mon client directed command retry = 5 2024-03-27T03:04:54.281 INFO:teuthology.orchestra.run.smithi196.stdout: mon election default strategy = 1 2024-03-27T03:04:54.281 INFO:teuthology.orchestra.run.smithi196.stdout: ms bind msgr2 = False 2024-03-27T03:04:54.281 INFO:teuthology.orchestra.run.smithi196.stdout: ms inject socket failures = 5000 2024-03-27T03:04:54.281 INFO:teuthology.orchestra.run.smithi196.stdout: ms type = async 2024-03-27T03:04:54.281 INFO:teuthology.orchestra.run.smithi196.stdout: fsid = 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:04:54.281 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.281 INFO:teuthology.orchestra.run.smithi196.stdout:[osd] 2024-03-27T03:04:54.282 INFO:teuthology.orchestra.run.smithi196.stdout: osd journal size = 100 2024-03-27T03:04:54.282 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.282 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub load threshold = 5.0 2024-03-27T03:04:54.282 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub max interval = 600 2024-03-27T03:04:54.282 INFO:teuthology.orchestra.run.smithi196.stdout: osd mclock profile = high_recovery_ops 2024-03-27T03:04:54.282 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.282 INFO:teuthology.orchestra.run.smithi196.stdout: osd recover clone overlap = true 2024-03-27T03:04:54.282 INFO:teuthology.orchestra.run.smithi196.stdout: osd recovery max chunk = 1048576 2024-03-27T03:04:54.282 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.282 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug shutdown = true 2024-03-27T03:04:54.282 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug op order = true 2024-03-27T03:04:54.282 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify stray on activate = true 2024-03-27T03:04:54.283 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug trim objects = true 2024-03-27T03:04:54.283 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.283 INFO:teuthology.orchestra.run.smithi196.stdout: osd open classes on start = true 2024-03-27T03:04:54.283 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug pg log writeout = true 2024-03-27T03:04:54.283 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.283 INFO:teuthology.orchestra.run.smithi196.stdout: osd deep scrub update digest min age = 30 2024-03-27T03:04:54.283 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.283 INFO:teuthology.orchestra.run.smithi196.stdout: osd map max advance = 10 2024-03-27T03:04:54.283 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.283 INFO:teuthology.orchestra.run.smithi196.stdout: journal zero on create = true 2024-03-27T03:04:54.283 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.283 INFO:teuthology.orchestra.run.smithi196.stdout: filestore ondisk finisher threads = 3 2024-03-27T03:04:54.284 INFO:teuthology.orchestra.run.smithi196.stdout: filestore apply finisher threads = 3 2024-03-27T03:04:54.284 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.284 INFO:teuthology.orchestra.run.smithi196.stdout: bdev debug aio = true 2024-03-27T03:04:54.284 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug misdirected ops = true 2024-03-27T03:04:54.284 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore block size = 96636764160 2024-03-27T03:04:54.284 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore compression algorithm = snappy 2024-03-27T03:04:54.284 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore compression mode = aggressive 2024-03-27T03:04:54.284 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore fsck on mount = True 2024-03-27T03:04:54.284 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore zero block detection = True 2024-03-27T03:04:54.284 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluefs = 20 2024-03-27T03:04:54.284 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluestore = 20 2024-03-27T03:04:54.284 INFO:teuthology.orchestra.run.smithi196.stdout: debug monc = 20 2024-03-27T03:04:54.285 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-27T03:04:54.285 INFO:teuthology.orchestra.run.smithi196.stdout: debug osd = 20 2024-03-27T03:04:54.285 INFO:teuthology.orchestra.run.smithi196.stdout: debug rocksdb = 10 2024-03-27T03:04:54.285 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd backfillfull_ratio = 0.85 2024-03-27T03:04:54.285 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd full ratio = 0.9 2024-03-27T03:04:54.285 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd nearfull ratio = 0.8 2024-03-27T03:04:54.285 INFO:teuthology.orchestra.run.smithi196.stdout: osd class default list = * 2024-03-27T03:04:54.285 INFO:teuthology.orchestra.run.smithi196.stdout: osd class load list = * 2024-03-27T03:04:54.285 INFO:teuthology.orchestra.run.smithi196.stdout: osd client watch timeout = 120 2024-03-27T03:04:54.285 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify cached snaps = True 2024-03-27T03:04:54.285 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify missing on start = True 2024-03-27T03:04:54.285 INFO:teuthology.orchestra.run.smithi196.stdout: osd failsafe full ratio = 0.95 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout: osd mclock override recovery settings = True 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout: osd objectstore = bluestore 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout: osd op queue = debug_random 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout: osd op queue cut off = debug_random 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout: osd shutdown pgref assert = True 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr] 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout: debug mgr = 20 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-03-27T03:04:54.286 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-03-27T03:04:54.287 INFO:teuthology.orchestra.run.smithi196.stdout: mgr/telemetry/nag = false 2024-03-27T03:04:54.287 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.287 INFO:teuthology.orchestra.run.smithi196.stdout:[mon] 2024-03-27T03:04:54.287 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-27T03:04:54.287 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-03-27T03:04:54.287 INFO:teuthology.orchestra.run.smithi196.stdout: debug paxos = 20 2024-03-27T03:04:54.287 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-03-27T03:04:54.287 INFO:teuthology.orchestra.run.smithi196.stdout: mon data avail warn = 5 2024-03-27T03:04:54.287 INFO:teuthology.orchestra.run.smithi196.stdout: mon mgr mkfs grace = 240 2024-03-27T03:04:54.287 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-03-27T03:04:54.287 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd reporter subtree level = osd 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd prime pg temp = true 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: auth mon ticket ttl = 660 # 11m 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: auth service ticket ttl = 240 # 4m 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: # don't complain about insecure global_id in the test suite 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: # 1m isn't quite enough 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: mon_down_mkfs_grace = 2m 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_filestore_osds = false 2024-03-27T03:04:54.288 INFO:teuthology.orchestra.run.smithi196.stdout: mon min osdmap epochs = 50 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune interval = 2 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune min = 15 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune txsize = 2 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: mon scrub interval = 300 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on pool no app = False 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: paxos service trim min = 10 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout:[client] 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: rgw cache enabled = true 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable ops log = true 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable usage log = true 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-27T03:04:54.289 INFO:teuthology.orchestra.run.smithi196.stdout: debug monc = 20 2024-03-27T03:04:54.290 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-27T03:04:54.290 INFO:teuthology.orchestra.run.smithi196.stdout: debug objecter = 20 2024-03-27T03:04:54.290 INFO:teuthology.orchestra.run.smithi196.stdout: debug rados = 20 2024-03-27T03:04:54.290 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.a] 2024-03-27T03:04:54.290 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.c] 2024-03-27T03:04:54.290 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.b] 2024-03-27T03:04:54.291 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-27T03:04:54.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-27T03:04:54.388 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-27T03:04:54.388 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:04:54.388 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-27T03:04:54.403 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:04:54.403 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-27T03:04:54.459 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2024-03-27T03:04:54.459 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:04:54.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-27T03:04:54.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T03:04:54.538 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:04:54.538 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:04:54.592 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi196.front.sepia.ceph.com 2024-03-27T03:04:54.592 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:04:54.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-27T03:04:54.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T03:04:54.632 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:04:54.632 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:04:54.686 INFO:tasks.ceph:Setting up mon nodes... 2024-03-27T03:04:54.687 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-27T03:04:54.687 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-27T03:04:54.742 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-27T03:04:54.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-27T03:04:54.806 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-27T03:04:54.808 INFO:tasks.ceph:Setting up mds nodes... 2024-03-27T03:04:54.809 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-27T03:04:54.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-27T03:04:54.854 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-27T03:04:54.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-27T03:04:54.910 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-27T03:04:54.925 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-27T03:04:54.925 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-27T03:04:54.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-27T03:04:54.952 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T03:04:54.952 INFO:tasks.ceph:role: osd.0 2024-03-27T03:04:54.952 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-27T03:04:54.952 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-27T03:04:55.027 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T03:04:55.027 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T03:04:55.027 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T03:04:55.027 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T03:04:55.027 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T03:04:55.027 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-27T03:04:55.027 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T03:04:55.027 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T03:04:55.027 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T03:04:55.027 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T03:04:55.528 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-27T03:04:55.530 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-27T03:04:55.530 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-27T03:04:55.678 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-27T03:04:55.711 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-27T03:04:55.778 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T03:04:55.779 INFO:tasks.ceph:role: osd.1 2024-03-27T03:04:55.779 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-27T03:04:55.779 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-27T03:04:55.854 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T03:04:55.854 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T03:04:55.854 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T03:04:55.854 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T03:04:55.854 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T03:04:55.855 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-27T03:04:55.855 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T03:04:55.855 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T03:04:55.855 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T03:04:55.855 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T03:04:56.338 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-27T03:04:56.340 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-27T03:04:56.340 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-27T03:04:56.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-27T03:04:56.496 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-27T03:04:56.564 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T03:04:56.564 INFO:tasks.ceph:role: osd.2 2024-03-27T03:04:56.564 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-27T03:04:56.564 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-27T03:04:56.640 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T03:04:56.640 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T03:04:56.640 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T03:04:56.641 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T03:04:56.641 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T03:04:56.641 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-27T03:04:56.641 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T03:04:56.641 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T03:04:56.641 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T03:04:56.641 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T03:04:57.174 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-27T03:04:57.176 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-27T03:04:57.176 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-27T03:04:57.323 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-27T03:04:57.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-27T03:04:57.424 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T03:04:57.424 INFO:tasks.ceph:role: osd.3 2024-03-27T03:04:57.424 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-27T03:04:57.424 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-27T03:04:57.498 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T03:04:57.498 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T03:04:57.498 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T03:04:57.498 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T03:04:57.498 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T03:04:57.498 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-27T03:04:57.498 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T03:04:57.498 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T03:04:57.498 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T03:04:57.498 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T03:04:57.995 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-27T03:04:57.997 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-27T03:04:57.998 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-27T03:04:58.081 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-27T03:04:58.157 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:04:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:04:58.267+0000 7fcad6e11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-27T03:04:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:04:58.267+0000 7fcad6e11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-27T03:04:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:04:58.267+0000 7fcad6e11640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-27T03:04:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:04:58.267+0000 7fcad6e11640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-27T03:04:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:04:58.267+0000 7fcad6e11640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-27T03:04:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:04:58.268+0000 7fcad6e11640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-27T03:05:01.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-27T03:05:01.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:05:01.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:01.242+0000 7f88ba860640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-27T03:05:01.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:01.242+0000 7f88ba860640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-27T03:05:01.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:01.242+0000 7f88ba860640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-27T03:05:01.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:01.242+0000 7f88ba860640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-27T03:05:01.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:01.242+0000 7f88ba860640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-27T03:05:01.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:01.243+0000 7f88ba860640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-27T03:05:04.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-27T03:05:04.128 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:05:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:04.226+0000 7f8d5d219640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-27T03:05:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:04.226+0000 7f8d5d219640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-27T03:05:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:04.226+0000 7f8d5d219640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-27T03:05:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:04.226+0000 7f8d5d219640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-27T03:05:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:04.226+0000 7f8d5d219640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-27T03:05:04.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:04.227+0000 7f8d5d219640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-27T03:05:07.074 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-27T03:05:07.106 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:05:07.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:07.201+0000 7f3fd6613640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-27T03:05:07.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:07.202+0000 7f3fd6613640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-27T03:05:07.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:07.202+0000 7f3fd6613640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-27T03:05:07.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:07.202+0000 7f3fd6613640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-27T03:05:07.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:07.202+0000 7f3fd6613640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-27T03:05:07.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:07.202+0000 7f3fd6613640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-27T03:05:10.049 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-27T03:05:10.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-27T03:05:10.110 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T03:05:10.110 INFO:tasks.ceph:role: osd.4 2024-03-27T03:05:10.110 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-27T03:05:10.110 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-27T03:05:10.186 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T03:05:10.186 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T03:05:10.187 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T03:05:10.187 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T03:05:10.187 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T03:05:10.187 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-27T03:05:10.187 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T03:05:10.187 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T03:05:10.187 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T03:05:10.187 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T03:05:10.694 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-27T03:05:10.695 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-27T03:05:10.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-27T03:05:10.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-27T03:05:10.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-27T03:05:10.944 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T03:05:10.944 INFO:tasks.ceph:role: osd.5 2024-03-27T03:05:10.945 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-27T03:05:10.945 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-27T03:05:11.020 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T03:05:11.020 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T03:05:11.020 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T03:05:11.020 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T03:05:11.020 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T03:05:11.020 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-27T03:05:11.020 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T03:05:11.020 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T03:05:11.020 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T03:05:11.021 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T03:05:11.462 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-27T03:05:11.464 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-27T03:05:11.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-27T03:05:11.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-27T03:05:11.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-27T03:05:11.656 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T03:05:11.656 INFO:tasks.ceph:role: osd.6 2024-03-27T03:05:11.656 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-27T03:05:11.657 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-27T03:05:11.731 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T03:05:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T03:05:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T03:05:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T03:05:11.731 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T03:05:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-27T03:05:11.732 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T03:05:11.732 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T03:05:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T03:05:11.732 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T03:05:12.228 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-27T03:05:12.230 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-27T03:05:12.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-27T03:05:12.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-27T03:05:12.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-27T03:05:12.477 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T03:05:12.477 INFO:tasks.ceph:role: osd.7 2024-03-27T03:05:12.477 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-27T03:05:12.477 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-27T03:05:12.552 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T03:05:12.552 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T03:05:12.552 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T03:05:12.552 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T03:05:12.552 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T03:05:12.552 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-27T03:05:12.552 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T03:05:12.552 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T03:05:12.552 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T03:05:12.552 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T03:05:13.064 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-27T03:05:13.065 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-27T03:05:13.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-27T03:05:13.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-27T03:05:13.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:05:13.339 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:13.346+0000 7f7e3b060640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-27T03:05:13.340 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:13.347+0000 7f7e3b060640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-27T03:05:13.340 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:13.347+0000 7f7e3b060640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-27T03:05:13.340 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:13.347+0000 7f7e3b060640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-27T03:05:13.340 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:13.347+0000 7f7e3b060640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-27T03:05:13.340 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:13.347+0000 7f7e3b060640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-27T03:05:16.182 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-27T03:05:16.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:05:16.297 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:16.304+0000 7f11431bd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-27T03:05:16.298 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:16.304+0000 7f11431bd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-27T03:05:16.298 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:16.304+0000 7f11431bd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-27T03:05:16.298 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:16.305+0000 7f11431bd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-27T03:05:16.298 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:16.305+0000 7f11431bd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-27T03:05:16.298 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:16.305+0000 7f11431bd640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-27T03:05:19.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-27T03:05:19.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:05:19.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:19.265+0000 7f9a32e0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-27T03:05:19.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:19.266+0000 7f9a32e0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-27T03:05:19.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:19.266+0000 7f9a32e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-27T03:05:19.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:19.266+0000 7f9a32e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-27T03:05:19.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:19.266+0000 7f9a32e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-27T03:05:19.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:19.266+0000 7f9a32e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-27T03:05:22.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-27T03:05:22.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:05:22.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:22.226+0000 7fac92c22640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-27T03:05:22.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:22.227+0000 7fac92c22640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-27T03:05:22.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:22.227+0000 7fac92c22640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-27T03:05:22.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:22.227+0000 7fac92c22640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-27T03:05:22.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:22.227+0000 7fac92c22640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-27T03:05:22.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:22.227+0000 7fac92c22640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-27T03:05:25.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-27T03:05:25.103 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-27T03:05:25.104 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:05:25.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-27T03:05:25.135 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:05:25.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-27T03:05:25.206 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:05:25.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-27T03:05:25.277 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:05:25.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-27T03:05:25.348 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:05:25.348 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-27T03:05:25.418 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:05:25.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-27T03:05:25.450 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:05:25.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-27T03:05:25.519 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:05:25.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-27T03:05:25.589 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:05:25.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-27T03:05:25.658 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:05:25.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-27T03:05:25.727 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:05:25.727 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-27T03:05:25.745 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:05:25.745 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-27T03:05:25.784 INFO:tasks.ceph:Adding keys to all mons... 2024-03-27T03:05:25.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-27T03:05:25.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-27T03:05:25.828 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.y] 2024-03-27T03:05:25.828 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDWjANmumW7LBAApXse0X0yHBxA7uoHGMenUQ== 2024-03-27T03:05:25.828 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDajANm1qL5DxAAEU9wnnSFFIf4iEKqjGg7BQ== 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDdjANml7h9DhAAhgH9ytqnAOas8P7zXBmAQg== 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDgjANmJoqNDRAA7EYAyYhIhm9leCYGc+/d1A== 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDjjANm0e4UDBAA/tTK5Gdkp1H9CTBVqWU/xA== 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDWjANmrx2MMBAAEio44ClWwp46izyRcwjdaA== 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.4] 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDpjANmxQq6FBAAsAKMr3ArjtgKvwhZYMIt/w== 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.5] 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDsjANmpHE4EhAALF1pZSGCQXf3qGUhqDhtCw== 2024-03-27T03:05:25.829 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.6] 2024-03-27T03:05:25.830 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDvjANm1QvnDxAA4UR0vw13uoYCqkQ2ZtvYfQ== 2024-03-27T03:05:25.830 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.7] 2024-03-27T03:05:25.830 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDyjANmswKUDRAAeZq7FvqecXCQZuqVLGUfrA== 2024-03-27T03:05:25.830 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-03-27T03:05:25.830 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDWjANmlFJAMxAAyG4s8csWpLmN5vZXL17f0A== 2024-03-27T03:05:25.830 INFO:teuthology.orchestra.run.smithi153.stdout:[client.1] 2024-03-27T03:05:25.830 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDWjANmAGnANhAA9raJ+GwKtHNcY9/yWF2dfA== 2024-03-27T03:05:25.851 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.y] 2024-03-27T03:05:25.851 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDWjANmumW7LBAApXse0X0yHBxA7uoHGMenUQ== 2024-03-27T03:05:25.851 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.0] 2024-03-27T03:05:25.851 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDajANm1qL5DxAAEU9wnnSFFIf4iEKqjGg7BQ== 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.1] 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDdjANml7h9DhAAhgH9ytqnAOas8P7zXBmAQg== 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.2] 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDgjANmJoqNDRAA7EYAyYhIhm9leCYGc+/d1A== 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.3] 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDjjANm0e4UDBAA/tTK5Gdkp1H9CTBVqWU/xA== 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.x] 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDWjANmrx2MMBAAEio44ClWwp46izyRcwjdaA== 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.4] 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDpjANmxQq6FBAAsAKMr3ArjtgKvwhZYMIt/w== 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.5] 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDsjANmpHE4EhAALF1pZSGCQXf3qGUhqDhtCw== 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.6] 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDvjANm1QvnDxAA4UR0vw13uoYCqkQ2ZtvYfQ== 2024-03-27T03:05:25.852 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.7] 2024-03-27T03:05:25.853 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDyjANmswKUDRAAeZq7FvqecXCQZuqVLGUfrA== 2024-03-27T03:05:25.853 INFO:teuthology.orchestra.run.smithi196.stdout:[client.0] 2024-03-27T03:05:25.853 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDWjANmlFJAMxAAyG4s8csWpLmN5vZXL17f0A== 2024-03-27T03:05:25.853 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2024-03-27T03:05:25.853 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDWjANmAGnANhAA9raJ+GwKtHNcY9/yWF2dfA== 2024-03-27T03:05:25.854 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T03:05:25.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T03:05:25.948 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:25.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.110 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.163 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T03:05:26.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T03:05:26.217 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.271 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.324 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.377 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T03:05:26.432 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T03:05:26.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T03:05:26.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T03:05:26.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T03:05:26.546 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-27T03:05:26.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-27T03:05:26.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-27T03:05:26.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-27T03:05:26.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-27T03:05:26.765 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-27T03:05:26.862 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-27T03:05:26.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-27T03:05:26.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-27T03:05:27.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-27T03:05:27.042 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:05:27.045 DEBUG:teuthology.orchestra.run.smithi196:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-27T03:05:27.097 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-27T03:05:27.097 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-27T03:05:27.097 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-27T03:05:27.100 INFO:tasks.ceph.mon.a:Started 2024-03-27T03:05:27.100 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-27T03:05:27.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-27T03:05:27.102 INFO:tasks.ceph.mon.c:Started 2024-03-27T03:05:27.103 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-27T03:05:27.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-27T03:05:27.139 INFO:tasks.ceph.mon.b:Started 2024-03-27T03:05:27.139 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-27T03:05:27.139 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-27T03:05:27.139 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-27T03:05:27.142 INFO:tasks.ceph.mgr.y:Started 2024-03-27T03:05:27.142 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-27T03:05:27.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-27T03:05:27.144 INFO:tasks.ceph.mgr.x:Started 2024-03-27T03:05:27.144 INFO:tasks.ceph:Setting crush tunables to default 2024-03-27T03:05:27.144 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd crush tunables default 2024-03-27T03:05:27.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.252+0000 7f80d4f15640 10 monclient: get_monmap_and_config 2024-03-27T03:05:27.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.252+0000 7f80d4f15640 10 monclient: build_initial_monmap 2024-03-27T03:05:27.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.252+0000 7f80d4f15640 10 monclient: monmap: 2024-03-27T03:05:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-27T03:05:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-27T03:05:27.253574+0000 2024-03-27T03:05:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-27T03:05:27.253574+0000 2024-03-27T03:05:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:27.248 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.253+0000 7f80d4f15640 1 Processor -- start 2024-03-27T03:05:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.253+0000 7f80d4f15640 1 -- start start 2024-03-27T03:05:27.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.253+0000 7f80d4f15640 10 monclient: init 2024-03-27T03:05:27.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.253+0000 7f80d4f15640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:27.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.253+0000 7f80d4f15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:05:27.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.254+0000 7f80d4f15640 10 monclient(hunting): picked mon.noname-b con 0x7f80d009ac40 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:27.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.254+0000 7f80d4f15640 10 monclient(hunting): picked mon.noname-c con 0x7f80d009c870 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:27.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.254+0000 7f80d4f15640 10 monclient(hunting): picked mon.noname-a con 0x7f80d00b4360 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:27.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.254+0000 7f80d4f15640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d009dec0 con 0x7f80d00b4360 2024-03-27T03:05:27.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.254+0000 7f80d4f15640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d009da30 con 0x7f80d009c870 2024-03-27T03:05:27.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.254+0000 7f80d4f15640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d00b39c0 con 0x7f80d009ac40 2024-03-27T03:05:27.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.254+0000 7f80d4f15640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:27.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.254+0000 7f80d4f15640 10 monclient(hunting): authenticate will time out at 1287.559630s 2024-03-27T03:05:27.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.254+0000 7f80ce59c640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f80d009ac40 0x7f80d00b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:56316/0 (socket says 172.21.15.153:56316) 2024-03-27T03:05:27.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:27.254+0000 7f80ce59c640 1 -- 172.21.15.153:0/1170060231 learned_addr learned my addr 172.21.15.153:0/1170060231 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:05:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 10 monclient(hunting): tick 2024-03-27T03:05:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 10 monclient(hunting): _check_auth_tickets 2024-03-27T03:05:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 1 monclient(hunting): continuing hunt 2024-03-27T03:05:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-27T03:05:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 1 -- 172.21.15.153:0/1170060231 >> v1:172.21.15.153:6790/0 conn(0x7f80d009ac40 legacy=0x7f80d00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:30.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 1 -- 172.21.15.153:0/1170060231 >> v1:172.21.15.196:6789/0 conn(0x7f80d009c870 legacy=0x7f80d013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 1 -- 172.21.15.153:0/1170060231 >> v1:172.21.15.153:6789/0 conn(0x7f80d00b4360 legacy=0x7f80d014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:05:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 10 monclient(hunting): picked mon.noname-a con 0x7f80b00036e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 10 monclient(hunting): picked mon.noname-c con 0x7f80b0007300 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 10 monclient(hunting): picked mon.noname-b con 0x7f80b000b020 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 1 -- 172.21.15.153:0/1170060231 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d00b5160 con 0x7f80b00036e0 2024-03-27T03:05:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 1 -- 172.21.15.153:0/1170060231 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b000fe30 con 0x7f80b0007300 2024-03-27T03:05:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 1 -- 172.21.15.153:0/1170060231 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b0011070 con 0x7f80b000b020 2024-03-27T03:05:30.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:30.254+0000 7f80bffff640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:32.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.459+0000 7f80cd59a640 1 -- 172.21.15.153:0/1170060231 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076727332 0 0) 0x7f80d00b5160 con 0x7f80b00036e0 2024-03-27T03:05:32.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.459+0000 7f80cd59a640 10 monclient(hunting): _negotiate 2024-03-27T03:05:32.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.459+0000 7f80cd59a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:32.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.459+0000 7f80cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:32.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.459+0000 7f80cd59a640 10 monclient(hunting): my global_id is 4101 2024-03-27T03:05:32.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.459+0000 7f80cd59a640 1 -- 172.21.15.153:0/1170060231 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b8003520 con 0x7f80b00036e0 2024-03-27T03:05:32.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.459+0000 7f80cd59a640 1 -- 172.21.15.153:0/1170060231 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704303443 0 0) 0x7f80b0011070 con 0x7f80b000b020 2024-03-27T03:05:32.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.459+0000 7f80cd59a640 10 monclient(hunting): _negotiate 2024-03-27T03:05:32.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.459+0000 7f80cd59a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:32.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.459+0000 7f80cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:32.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.459+0000 7f80cd59a640 10 monclient(hunting): my global_id is 4106 2024-03-27T03:05:32.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.459+0000 7f80cd59a640 1 -- 172.21.15.153:0/1170060231 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d00b5160 con 0x7f80b000b020 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 1 -- 172.21.15.153:0/1170060231 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 75584794 0 0) 0x7f80d00b5160 con 0x7f80b000b020 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 1 -- 172.21.15.153:0/1170060231 >> v1:172.21.15.196:6789/0 conn(0x7f80b0007300 legacy=0x7f80b0009790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 1 -- 172.21.15.153:0/1170060231 >> v1:172.21.15.153:6789/0 conn(0x7f80b00036e0 legacy=0x7f80b0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 10 monclient: _finish_hunting 0 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 1 monclient: found mon.noname-b 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 1 -- 172.21.15.153:0/1170060231 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b00122b0 con 0x7f80b000b020 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 10 monclient: _finish_auth 0 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 10 monclient: _check_auth_tickets 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 1 -- 172.21.15.153:0/1170060231 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f80c00017f0 con 0x7f80b000b020 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 10 monclient: dump: 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80d4f15640 5 monclient: authenticate success, global_id 4106 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80d4f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 1 -- 172.21.15.153:0/1170060231 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80c0001950 con 0x7f80b000b020 2024-03-27T03:05:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 1 -- 172.21.15.153:0/1170060231 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f80c0001d60 con 0x7f80b000b020 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80cd59a640 10 monclient: dump: 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:32.457 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:32.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80d4f15640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:32.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.460+0000 7f80d4f15640 4 set_mon_vals no callback set 2024-03-27T03:05:32.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.461+0000 7f80d4f15640 10 monclient: shutdown 2024-03-27T03:05:32.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.461+0000 7f80d4f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:32.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.461+0000 7f80d4f15640 1 -- 172.21.15.153:0/1170060231 >> v1:172.21.15.153:6790/0 conn(0x7f80b000b020 legacy=0x7f80b000d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:32.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.461+0000 7f80d4f15640 1 -- 172.21.15.153:0/1170060231 shutdown_connections 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.461+0000 7f80d4f15640 1 -- 172.21.15.153:0/1170060231 >> 172.21.15.153:0/1170060231 conn(0x7f80d0066b00 msgr2=0x7f80d0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.461+0000 7f80d4f15640 1 -- 172.21.15.153:0/1170060231 shutdown_connections 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.462+0000 7f80d4f15640 1 -- 172.21.15.153:0/1170060231 wait complete. 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.463+0000 7f80d4f15640 10 monclient: build_initial_monmap 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.463+0000 7f80d4f15640 10 monclient: monmap: 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:32.464 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.463+0000 7f80d4f15640 1 librados: starting msgr at 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.463+0000 7f80d4f15640 1 librados: starting objecter 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.463+0000 7f80d4f15640 1 Processor -- start 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.463+0000 7f80d4f15640 1 -- start start 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.463+0000 7f80d4f15640 1 librados: setting wanted keys 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.463+0000 7f80d4f15640 1 librados: calling monclient init 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.463+0000 7f80d4f15640 10 monclient: init 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80d4f15640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80d4f15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80d4f15640 10 monclient(hunting): picked mon.noname-a con 0x7f80b00036e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80d4f15640 10 monclient(hunting): picked mon.noname-b con 0x7f80b0007300 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80d4f15640 10 monclient(hunting): picked mon.noname-c con 0x7f80b000b020 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80d4f15640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d0145cb0 con 0x7f80b00036e0 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80d4f15640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d0145f90 con 0x7f80b0007300 2024-03-27T03:05:32.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80d4f15640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d01da8a0 con 0x7f80b000b020 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80d4f15640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80d4f15640 10 monclient(hunting): authenticate will time out at 1292.769917s 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80ce59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f80b00036e0 0x7f80d0144be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50674/0 (socket says 172.21.15.153:50674) 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80ce59c640 1 -- 172.21.15.153:0/593342825 learned_addr learned my addr 172.21.15.153:0/593342825 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80beffd640 10 client.?.objecter ms_handle_connect 0x7f80b00036e0 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80beffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80beffd640 10 client.?.objecter ms_handle_connect 0x7f80b000b020 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80beffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80beffd640 10 client.?.objecter ms_handle_connect 0x7f80b0007300 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.464+0000 7f80beffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461169850 0 0) 0x7f80d0145f90 con 0x7f80b0007300 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:32.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 10 monclient(hunting): my global_id is 4102 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b0006a20 con 0x7f80b0007300 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1466930791 0 0) 0x7f80b0006a20 con 0x7f80b0007300 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80d0145f90 con 0x7f80b0007300 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f80c4001840 con 0x7f80b0007300 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 10 monclient(hunting): dump: 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:32.467 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.465+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1108903530 0 0) 0x7f80d0145f90 con 0x7f80b0007300 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 >> v1:172.21.15.153:6790/0 conn(0x7f80b000b020 legacy=0x7f80d00b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 >> v1:172.21.15.153:6789/0 conn(0x7f80b00036e0 legacy=0x7f80d0144be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 10 monclient: _finish_hunting 0 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 1 monclient: found mon.b 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d0140f50 con 0x7f80b0007300 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 10 monclient: _finish_auth 0 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 10 monclient: _check_auth_tickets 2024-03-27T03:05:32.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80d4f15640 5 monclient: authenticate success, global_id 4102 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80d4f15640 10 monclient: _renew_subs 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80d4f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80d4f15640 1 -- 172.21.15.153:0/593342825 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d01db4d0 con 0x7f80b0007300 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80d4f15640 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80d4f15640 10 monclient: _renew_subs 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80d4f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80d4f15640 1 -- 172.21.15.153:0/593342825 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d01441f0 con 0x7f80b0007300 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80d4f15640 1 librados: init done 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80c40035f0 con 0x7f80b0007300 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f80c4003900 con 0x7f80b0007300 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80ccd99640 4 set_mon_vals no callback set 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 10 monclient: dump: 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f80c4003e70 con 0x7f80b0007300 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 552737362 0 0) 0x7f80c40053f0 con 0x7f80b0007300 2024-03-27T03:05:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 10 client.4102.objecter ms_dispatch 0x7f80d0000b90 osd_map(1..1 src has 1..1) v4 2024-03-27T03:05:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-03-27T03:05:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-03-27T03:05:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.466+0000 7f80beffd640 20 client.4102.objecter dump_active .. 0 homeless 2024-03-27T03:05:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.467+0000 7f80d4f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.467+0000 7f80d4f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.467+0000 7f80d4f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.467+0000 7f80d4f15640 1 -- 172.21.15.153:0/593342825 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b4005080 con 0x7f80b0007300 2024-03-27T03:05:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.470+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f80c4004410 con 0x7f80b0007300 2024-03-27T03:05:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.470+0000 7f80beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.470+0000 7f80beffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:32.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.545+0000 7f80d4f15640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-27T03:05:32.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.545+0000 7f80d4f15640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-27T03:05:32.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.545+0000 7f80d4f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:32.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.545+0000 7f80d4f15640 1 -- 172.21.15.153:0/593342825 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f80b4005330 con 0x7f80b0007300 2024-03-27T03:05:32.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.549+0000 7f80beffd640 1 -- 172.21.15.153:0/593342825 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f80c4004690 con 0x7f80b0007300 2024-03-27T03:05:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.549+0000 7f80beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-27T03:05:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.549+0000 7f80beffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-27T03:05:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2024-03-27T03:05:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.549+0000 7f80d4f15640 10 librados: watch_flush enter 2024-03-27T03:05:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.549+0000 7f80d4f15640 10 librados: watch_flush exit 2024-03-27T03:05:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.549+0000 7f80d4f15640 20 client.4102.objecter shutdown clearing up homeless session... 2024-03-27T03:05:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.549+0000 7f80d4f15640 10 client.4102.objecter successfully canceled tick 2024-03-27T03:05:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.550+0000 7f80d4f15640 10 monclient: shutdown 2024-03-27T03:05:32.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.550+0000 7f80d4f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.550+0000 7f80d4f15640 1 -- 172.21.15.153:0/593342825 >> v1:172.21.15.196:6789/0 conn(0x7f80b0007300 legacy=0x7f80d01dadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.550+0000 7f80d4f15640 1 -- 172.21.15.153:0/593342825 shutdown_connections 2024-03-27T03:05:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.550+0000 7f80d4f15640 1 -- 172.21.15.153:0/593342825 >> 172.21.15.153:0/593342825 conn(0x7f80d0066b00 msgr2=0x7f80d013c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.550+0000 7f80d4f15640 1 -- 172.21.15.153:0/593342825 shutdown_connections 2024-03-27T03:05:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.550+0000 7f80d4f15640 1 -- 172.21.15.153:0/593342825 wait complete. 2024-03-27T03:05:32.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:32.550+0000 7f80d4f15640 1 librados: shutdown 2024-03-27T03:05:32.553 INFO:tasks.ceph:check_enable_crimson: False 2024-03-27T03:05:32.554 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-27T03:05:32.554 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:05:32.554 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-27T03:05:32.582 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:05:32.582 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-27T03:05:32.648 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:05:32.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-27T03:05:32.716 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-27T03:05:32.716 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-27T03:05:32.783 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:05:32.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-27T03:05:32.810 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:05:32.810 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-27T03:05:32.877 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:05:32.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-27T03:05:32.938 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:32.945+0000 7f9854a42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-27T03:05:32.940 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:32.947+0000 7f76d4a21040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-27T03:05:32.943 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-27T03:05:32.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-27T03:05:32.993 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:33.000+0000 7f76d4a21040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-27T03:05:32.994 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:33.001+0000 7f9854a42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-27T03:05:33.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 1d21a393-d352-4f2e-bc5f-67ef3d73e26b 0 2024-03-27T03:05:33.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.158+0000 7f816b5e8640 10 monclient: get_monmap_and_config 2024-03-27T03:05:33.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.158+0000 7f816b5e8640 10 monclient: build_initial_monmap 2024-03-27T03:05:33.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.158+0000 7f816b5e8640 10 monclient: monmap: 2024-03-27T03:05:33.151 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:33.151 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.151 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:05:33.159034+0000 2024-03-27T03:05:33.151 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:05:33.159034+0000 2024-03-27T03:05:33.151 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:33.151 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.151 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:33.151 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:33.152 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:33.152 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.161 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.158+0000 7f816b5e8640 1 Processor -- start 2024-03-27T03:05:33.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.158+0000 7f816b5e8640 1 -- start start 2024-03-27T03:05:33.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.158+0000 7f816b5e8640 10 monclient: init 2024-03-27T03:05:33.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f816b5e8640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:33.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f816b5e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:05:33.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f816b5e8640 10 monclient(hunting): picked mon.noname-c con 0x7f816409ac40 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:33.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f816b5e8640 10 monclient(hunting): picked mon.noname-a con 0x7f816409c870 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:33.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f816b5e8640 10 monclient(hunting): picked mon.noname-b con 0x7f81640b4360 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:33.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f816b5e8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816409dec0 con 0x7f816409c870 2024-03-27T03:05:33.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f816b5e8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816409da30 con 0x7f816409ac40 2024-03-27T03:05:33.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f816b5e8640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81640b39c0 con 0x7f81640b4360 2024-03-27T03:05:33.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f816b5e8640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f816b5e8640 10 monclient(hunting): authenticate will time out at 814.478196s 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f8169384640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f816409ac40 0x7f81640b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34696/0 (socket says 172.21.15.196:34696) 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f8169384640 1 -- 172.21.15.196:0/3596996894 learned_addr learned my addr 172.21.15.196:0/3596996894 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291907552 0 0) 0x7f816409da30 con 0x7f816409ac40 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 10 monclient(hunting): my global_id is 4105 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8150003520 con 0x7f816409ac40 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055101808 0 0) 0x7f8160001110 con 0x7f816409c870 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 10 monclient(hunting): my global_id is 4104 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f816409da30 con 0x7f816409c870 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815768402 0 0) 0x7f81640b39c0 con 0x7f81640b4360 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.159+0000 7f815bfff640 10 monclient(hunting): my global_id is 4118 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8160001110 con 0x7f81640b4360 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 556570047 0 0) 0x7f8150003520 con 0x7f816409ac40 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 >> v1:172.21.15.153:6790/0 conn(0x7f81640b4360 legacy=0x7f816414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 >> v1:172.21.15.153:6789/0 conn(0x7f816409c870 legacy=0x7f816413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 10 monclient: _finish_hunting 0 2024-03-27T03:05:33.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 1 monclient: found mon.noname-c 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81640b5160 con 0x7f816409ac40 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 10 monclient: _finish_auth 0 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 10 monclient: _check_auth_tickets 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f816b5e8640 5 monclient: authenticate success, global_id 4105 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8154001690 con 0x7f816409ac40 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f816b5e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 10 monclient: dump: 2024-03-27T03:05:33.165 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f816b5e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8154001870 con 0x7f816409ac40 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:33.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f816b5e8640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 1 -- 172.21.15.196:0/3596996894 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8154001c80 con 0x7f816409ac40 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f816b5e8640 4 set_mon_vals no callback set 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f815bfff640 10 monclient: dump: 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f816b5e8640 10 monclient: shutdown 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f816b5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f816b5e8640 1 -- 172.21.15.196:0/3596996894 >> v1:172.21.15.196:6789/0 conn(0x7f816409ac40 legacy=0x7f81640b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f816b5e8640 1 -- 172.21.15.196:0/3596996894 shutdown_connections 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.160+0000 7f816b5e8640 1 -- 172.21.15.196:0/3596996894 >> 172.21.15.196:0/3596996894 conn(0x7f8164066b00 msgr2=0x7f8164066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.161+0000 7f816b5e8640 1 -- 172.21.15.196:0/3596996894 shutdown_connections 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.161+0000 7f816b5e8640 1 -- 172.21.15.196:0/3596996894 wait complete. 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.161+0000 7f816b5e8640 10 monclient: build_initial_monmap 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.161+0000 7f816b5e8640 10 monclient: monmap: 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:33.168 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.161+0000 7f816b5e8640 1 librados: starting msgr at 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.161+0000 7f816b5e8640 1 librados: starting objecter 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.161+0000 7f816b5e8640 1 Processor -- start 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.161+0000 7f816b5e8640 1 -- start start 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.161+0000 7f816b5e8640 1 librados: setting wanted keys 2024-03-27T03:05:33.169 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.161+0000 7f816b5e8640 1 librados: calling monclient init 2024-03-27T03:05:33.170 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.161+0000 7f816b5e8640 10 monclient: init 2024-03-27T03:05:33.170 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f816b5e8640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:33.170 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f816b5e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:05:33.170 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f816b5e8640 10 monclient(hunting): picked mon.noname-a con 0x7f816409ac40 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:33.170 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f816b5e8640 10 monclient(hunting): picked mon.noname-c con 0x7f81640b1280 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:33.170 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f816b5e8640 10 monclient(hunting): picked mon.noname-b con 0x7f81641d6ed0 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:33.170 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f816b5e8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81641422a0 con 0x7f816409ac40 2024-03-27T03:05:33.170 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f816b5e8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8164142f50 con 0x7f81641d6ed0 2024-03-27T03:05:33.170 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f816b5e8640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81641ccab0 con 0x7f81640b1280 2024-03-27T03:05:33.170 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f816b5e8640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:33.170 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f816b5e8640 10 monclient(hunting): authenticate will time out at 814.481226s 2024-03-27T03:05:33.170 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8169b85640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f81641d6ed0 0x7f816413adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34708/0 (socket says 172.21.15.196:34708) 2024-03-27T03:05:33.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8169b85640 1 -- 172.21.15.196:0/3680239880 learned_addr learned my addr 172.21.15.196:0/3680239880 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:33.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 client.?.objecter ms_handle_connect 0x7f81641d6ed0 2024-03-27T03:05:33.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:33.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 client.?.objecter ms_handle_connect 0x7f816409ac40 2024-03-27T03:05:33.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:33.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 client.?.objecter ms_handle_connect 0x7f81640b1280 2024-03-27T03:05:33.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:33.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081510308 0 0) 0x7f8164142f50 con 0x7f81641d6ed0 2024-03-27T03:05:33.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.171 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 monclient(hunting): my global_id is 4108 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8148004eb0 con 0x7f81641d6ed0 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256619911 0 0) 0x7f81641422a0 con 0x7f816409ac40 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 monclient(hunting): my global_id is 4107 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8164142f50 con 0x7f816409ac40 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617918180 0 0) 0x7f81641ccab0 con 0x7f81640b1280 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 10 monclient(hunting): my global_id is 4121 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.162+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81641422a0 con 0x7f81640b1280 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2552992445 0 0) 0x7f8148004eb0 con 0x7f81641d6ed0 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81641ccab0 con 0x7f81641d6ed0 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f815c001c70 con 0x7f81641d6ed0 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient(hunting): dump: 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.173 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2343463593 0 0) 0x7f8164142f50 con 0x7f816409ac40 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8148004eb0 con 0x7f816409ac40 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8154001b80 con 0x7f816409ac40 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient(hunting): dump: 2024-03-27T03:05:33.174 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3658773799 0 0) 0x7f81641422a0 con 0x7f81640b1280 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8164142f50 con 0x7f81640b1280 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8160001860 con 0x7f81640b1280 2024-03-27T03:05:33.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient(hunting): dump: 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 45039202 0 0) 0x7f8148004eb0 con 0x7f816409ac40 2024-03-27T03:05:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 >> v1:172.21.15.153:6790/0 conn(0x7f81640b1280 legacy=0x7f8164142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 >> v1:172.21.15.196:6789/0 conn(0x7f81641d6ed0 legacy=0x7f816413adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 monclient: found mon.a 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81641ccc40 con 0x7f816409ac40 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient: _finish_auth 0 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f816b5e8640 5 monclient: authenticate success, global_id 4107 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f816b5e8640 10 monclient: _renew_subs 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f816b5e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:05:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f816b5e8640 1 -- 172.21.15.196:0/3680239880 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816413b4d0 con 0x7f816409ac40 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8154001ce0 con 0x7f816409ac40 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8154004d90 con 0x7f816409ac40 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 monclient: dump: 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.178 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f816b5e8640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f816b5e8640 10 monclient: _renew_subs 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f815b7fe640 4 set_mon_vals no callback set 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f816b5e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f816b5e8640 1 -- 172.21.15.196:0/3680239880 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81641e3720 con 0x7f816409ac40 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f816b5e8640 1 librados: init done 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8154005280 con 0x7f816409ac40 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1096179140 0 0) 0x7f8154005870 con 0x7f816409ac40 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 10 client.4107.objecter ms_dispatch 0x7f8164000b90 osd_map(2..2 src has 1..2) v4 2024-03-27T03:05:33.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 3 client.4107.objecter handle_osd_map got epochs [2,2] > 0 2024-03-27T03:05:33.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 3 client.4107.objecter handle_osd_map decoding full epoch 2 2024-03-27T03:05:33.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.163+0000 7f8159ffb640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-27T03:05:33.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.164+0000 7f816b5e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:33.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.164+0000 7f816b5e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:33.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.164+0000 7f816b5e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:05:33.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.164+0000 7f816b5e8640 1 -- 172.21.15.196:0/3680239880 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81641cce50 con 0x7f816409ac40 2024-03-27T03:05:33.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.166+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8154005a30 con 0x7f816409ac40 2024-03-27T03:05:33.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.166+0000 7f8159ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:33.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.166+0000 7f8159ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:33.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.239+0000 7f816b5e8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1d21a393-d352-4f2e-bc5f-67ef3d73e26b", "id": 0}] 2024-03-27T03:05:33.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.239+0000 7f816b5e8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1d21a393-d352-4f2e-bc5f-67ef3d73e26b", "id": 0}] 2024-03-27T03:05:33.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.239+0000 7f816b5e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:05:33.232 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.239+0000 7f816b5e8640 1 -- 172.21.15.196:0/3680239880 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1d21a393-d352-4f2e-bc5f-67ef3d73e26b", "id": 0} v 0) v1 -- 0x7f816409b020 con 0x7f816409ac40 2024-03-27T03:05:33.233 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.240+0000 7f8159ffb640 1 -- 172.21.15.196:0/3680239880 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1d21a393-d352-4f2e-bc5f-67ef3d73e26b", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1665905215 0 732707515) 0x7f8154005dc0 con 0x7f816409ac40 2024-03-27T03:05:33.233 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.240+0000 7f8159ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1d21a393-d352-4f2e-bc5f-67ef3d73e26b", "id": 0}] 2024-03-27T03:05:33.233 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.240+0000 7f8159ffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:33.233 INFO:teuthology.orchestra.run.smithi196.stdout:0 2024-03-27T03:05:33.234 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.241+0000 7f816b5e8640 10 librados: watch_flush enter 2024-03-27T03:05:33.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.241+0000 7f816b5e8640 10 librados: watch_flush exit 2024-03-27T03:05:33.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.241+0000 7f816b5e8640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-27T03:05:33.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.241+0000 7f816b5e8640 10 client.4107.objecter successfully canceled tick 2024-03-27T03:05:33.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.241+0000 7f816b5e8640 10 monclient: shutdown 2024-03-27T03:05:33.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.241+0000 7f816b5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:33.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.241+0000 7f816b5e8640 1 -- 172.21.15.196:0/3680239880 >> v1:172.21.15.153:6789/0 conn(0x7f816409ac40 legacy=0x7f8164141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.241+0000 7f816b5e8640 1 -- 172.21.15.196:0/3680239880 shutdown_connections 2024-03-27T03:05:33.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.241+0000 7f816b5e8640 1 -- 172.21.15.196:0/3680239880 >> 172.21.15.196:0/3680239880 conn(0x7f8164066b00 msgr2=0x7f8164066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:33.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.241+0000 7f816b5e8640 1 -- 172.21.15.196:0/3680239880 shutdown_connections 2024-03-27T03:05:33.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.241+0000 7f816b5e8640 1 -- 172.21.15.196:0/3680239880 wait complete. 2024-03-27T03:05:33.235 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.241+0000 7f816b5e8640 1 librados: shutdown 2024-03-27T03:05:33.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new b1388fe9-4d83-4f69-a75a-3774e75e17c3 1 2024-03-27T03:05:33.344 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.350+0000 7f059db23640 10 monclient: get_monmap_and_config 2024-03-27T03:05:33.344 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.350+0000 7f059db23640 10 monclient: build_initial_monmap 2024-03-27T03:05:33.344 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.350+0000 7f059db23640 10 monclient: monmap: 2024-03-27T03:05:33.344 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:33.344 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.345 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:05:33.351879+0000 2024-03-27T03:05:33.345 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:05:33.351879+0000 2024-03-27T03:05:33.345 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:33.345 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.345 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:33.345 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:33.345 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:33.345 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.351+0000 7f059db23640 1 Processor -- start 2024-03-27T03:05:33.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.351+0000 7f059db23640 1 -- start start 2024-03-27T03:05:33.345 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.351+0000 7f059db23640 10 monclient: init 2024-03-27T03:05:33.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f059db23640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:33.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f059db23640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:05:33.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f059db23640 10 monclient(hunting): picked mon.noname-a con 0x7f059809ac40 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:33.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f059db23640 10 monclient(hunting): picked mon.noname-c con 0x7f059809c870 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:33.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f059db23640 10 monclient(hunting): picked mon.noname-b con 0x7f05980b4360 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:33.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f059db23640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059809dec0 con 0x7f059809ac40 2024-03-27T03:05:33.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f059db23640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059809da30 con 0x7f059809c870 2024-03-27T03:05:33.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f059db23640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05980b39c0 con 0x7f05980b4360 2024-03-27T03:05:33.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f059db23640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f059db23640 10 monclient(hunting): authenticate will time out at 814.671176s 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f058ffff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f059809c870 0x7f059813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34710/0 (socket says 172.21.15.196:34710) 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f058ffff640 1 -- 172.21.15.196:0/1089816882 learned_addr learned my addr 172.21.15.196:0/1089816882 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455197264 0 0) 0x7f059809da30 con 0x7f059809c870 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 10 monclient(hunting): my global_id is 4111 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0584003520 con 0x7f059809c870 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270356917 0 0) 0x7f0588001110 con 0x7f059809ac40 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 10 monclient(hunting): my global_id is 4110 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f059809da30 con 0x7f059809ac40 2024-03-27T03:05:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806273896 0 0) 0x7f05980b39c0 con 0x7f05980b4360 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 10 monclient(hunting): my global_id is 4124 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0588001110 con 0x7f05980b4360 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1870913304 0 0) 0x7f0584003520 con 0x7f059809c870 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.352+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 >> v1:172.21.15.153:6790/0 conn(0x7f05980b4360 legacy=0x7f059814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 >> v1:172.21.15.153:6789/0 conn(0x7f059809ac40 legacy=0x7f05980b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 1 monclient: found mon.noname-c 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05980b5160 con 0x7f059809c870 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 10 monclient: _finish_auth 0 2024-03-27T03:05:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0580001690 con 0x7f059809c870 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f059db23640 5 monclient: authenticate success, global_id 4111 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f059db23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 10 monclient: dump: 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.357 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f059db23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0580001870 con 0x7f059809c870 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f059db23640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 1 -- 172.21.15.196:0/1089816882 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0580001c80 con 0x7f059809c870 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f059db23640 4 set_mon_vals no callback set 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f0596ffd640 10 monclient: dump: 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.358 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f059db23640 10 monclient: shutdown 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f059db23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f059db23640 1 -- 172.21.15.196:0/1089816882 >> v1:172.21.15.196:6789/0 conn(0x7f059809c870 legacy=0x7f059813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f059db23640 1 -- 172.21.15.196:0/1089816882 shutdown_connections 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f059db23640 1 -- 172.21.15.196:0/1089816882 >> 172.21.15.196:0/1089816882 conn(0x7f0598066b00 msgr2=0x7f0598066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f059db23640 1 -- 172.21.15.196:0/1089816882 shutdown_connections 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.353+0000 7f059db23640 1 -- 172.21.15.196:0/1089816882 wait complete. 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.354+0000 7f059db23640 10 monclient: build_initial_monmap 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.354+0000 7f059db23640 10 monclient: monmap: 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:33.359 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.354+0000 7f059db23640 1 librados: starting msgr at 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.354+0000 7f059db23640 1 librados: starting objecter 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.354+0000 7f059db23640 1 Processor -- start 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.354+0000 7f059db23640 1 -- start start 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.354+0000 7f059db23640 1 librados: setting wanted keys 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.354+0000 7f059db23640 1 librados: calling monclient init 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.354+0000 7f059db23640 10 monclient: init 2024-03-27T03:05:33.360 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f059db23640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f059db23640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f059db23640 10 monclient(hunting): picked mon.noname-b con 0x7f059809ac40 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f059db23640 10 monclient(hunting): picked mon.noname-a con 0x7f05981d7bc0 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f059db23640 10 monclient(hunting): picked mon.noname-c con 0x7f05980b17f0 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f059db23640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059813b890 con 0x7f05981d7bc0 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f059db23640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05981d76a0 con 0x7f059809ac40 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f059db23640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05981406b0 con 0x7f05980b17f0 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f059db23640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f059db23640 10 monclient(hunting): authenticate will time out at 814.674054s 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f05977fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f059809ac40 0x7f059813ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34726/0 (socket says 172.21.15.196:34726) 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f05977fe640 1 -- 172.21.15.196:0/3653177394 learned_addr learned my addr 172.21.15.196:0/3653177394 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 client.?.objecter ms_handle_connect 0x7f059809ac40 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 client.?.objecter ms_handle_connect 0x7f05981d7bc0 2024-03-27T03:05:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933925208 0 0) 0x7f05981d76a0 con 0x7f059809ac40 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): my global_id is 4114 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0578004eb0 con 0x7f059809ac40 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 client.?.objecter ms_handle_connect 0x7f05980b17f0 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551527402 0 0) 0x7f059813b890 con 0x7f05981d7bc0 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): my global_id is 4113 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05981d76a0 con 0x7f05981d7bc0 2024-03-27T03:05:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451644132 0 0) 0x7f05981406b0 con 0x7f05980b17f0 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): my global_id is 4127 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f059813b890 con 0x7f05980b17f0 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4206635181 0 0) 0x7f0578004eb0 con 0x7f059809ac40 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05981406b0 con 0x7f059809ac40 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f05880019a0 con 0x7f059809ac40 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): dump: 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.363 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 401947616 0 0) 0x7f05981d76a0 con 0x7f05981d7bc0 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0578004eb0 con 0x7f05981d7bc0 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0580001b80 con 0x7f05981d7bc0 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): dump: 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1918111064 0 0) 0x7f059813b890 con 0x7f05980b17f0 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05981d76a0 con 0x7f05980b17f0 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0590001bd0 con 0x7f05980b17f0 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 10 monclient(hunting): dump: 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.365 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1976747818 0 0) 0x7f05981406b0 con 0x7f059809ac40 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.355+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 >> v1:172.21.15.153:6790/0 conn(0x7f05980b17f0 legacy=0x7f059813b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 >> v1:172.21.15.153:6789/0 conn(0x7f05981d7bc0 legacy=0x7f059813aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 1 monclient: found mon.b 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.366 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0598140840 con 0x7f059809ac40 2024-03-27T03:05:33.367 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 10 monclient: _finish_auth 0 2024-03-27T03:05:33.367 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:05:33.369 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:33.369 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 5 monclient: authenticate success, global_id 4114 2024-03-27T03:05:33.369 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 10 monclient: _renew_subs 2024-03-27T03:05:33.369 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.369 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 1 -- 172.21.15.196:0/3653177394 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05980b6f40 con 0x7f059809ac40 2024-03-27T03:05:33.369 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 10 monclient: _renew_subs 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 1 -- 172.21.15.196:0/3653177394 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05981d6f40 con 0x7f059809ac40 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 1 librados: init done 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0588001c40 con 0x7f059809ac40 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0588003a60 con 0x7f059809ac40 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 10 monclient: dump: 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.370 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f05967fc640 4 set_mon_vals no callback set 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0588003fd0 con 0x7f059809ac40 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 524461440 0 0) 0x7f0588006650 con 0x7f059809ac40 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 10 client.4114.objecter ms_dispatch 0x7f0598000b90 osd_map(3..3 src has 1..3) v4 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 3 client.4114.objecter handle_osd_map got epochs [3,3] > 0 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 3 client.4114.objecter handle_osd_map decoding full epoch 3 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f0594ff9640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:33.371 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.356+0000 7f059db23640 1 -- 172.21.15.196:0/3653177394 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057c005080 con 0x7f059809ac40 2024-03-27T03:05:33.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.358+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0588004460 con 0x7f059809ac40 2024-03-27T03:05:33.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.358+0000 7f0594ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:33.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.358+0000 7f0594ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:33.424 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.431+0000 7f059db23640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b1388fe9-4d83-4f69-a75a-3774e75e17c3", "id": 1}] 2024-03-27T03:05:33.424 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.431+0000 7f059db23640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b1388fe9-4d83-4f69-a75a-3774e75e17c3", "id": 1}] 2024-03-27T03:05:33.424 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.431+0000 7f059db23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.425 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.431+0000 7f059db23640 1 -- 172.21.15.196:0/3653177394 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b1388fe9-4d83-4f69-a75a-3774e75e17c3", "id": 1} v 0) v1 -- 0x7f057c0058b0 con 0x7f059809ac40 2024-03-27T03:05:33.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.435+0000 7f0594ff9640 1 -- 172.21.15.196:0/3653177394 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b1388fe9-4d83-4f69-a75a-3774e75e17c3", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3155094609 0 940483276) 0x7f0588005890 con 0x7f059809ac40 2024-03-27T03:05:33.427 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.435+0000 7f0594ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b1388fe9-4d83-4f69-a75a-3774e75e17c3", "id": 1}] 2024-03-27T03:05:33.428 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.435+0000 7f0594ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:33.428 INFO:teuthology.orchestra.run.smithi196.stdout:1 2024-03-27T03:05:33.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.435+0000 7f059db23640 10 librados: watch_flush enter 2024-03-27T03:05:33.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.435+0000 7f059db23640 10 librados: watch_flush exit 2024-03-27T03:05:33.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.435+0000 7f059db23640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-27T03:05:33.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.435+0000 7f059db23640 10 client.4114.objecter successfully canceled tick 2024-03-27T03:05:33.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.436+0000 7f059db23640 10 monclient: shutdown 2024-03-27T03:05:33.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.436+0000 7f059db23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:33.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.436+0000 7f059db23640 1 -- 172.21.15.196:0/3653177394 >> v1:172.21.15.196:6789/0 conn(0x7f059809ac40 legacy=0x7f059813ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.436+0000 7f059db23640 1 -- 172.21.15.196:0/3653177394 shutdown_connections 2024-03-27T03:05:33.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.436+0000 7f059db23640 1 -- 172.21.15.196:0/3653177394 >> 172.21.15.196:0/3653177394 conn(0x7f0598066b00 msgr2=0x7f059813d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:33.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.436+0000 7f059db23640 1 -- 172.21.15.196:0/3653177394 shutdown_connections 2024-03-27T03:05:33.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.436+0000 7f059db23640 1 -- 172.21.15.196:0/3653177394 wait complete. 2024-03-27T03:05:33.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.436+0000 7f059db23640 1 librados: shutdown 2024-03-27T03:05:33.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 2193eae0-0507-4a25-b242-eb5d61931b56 2 2024-03-27T03:05:33.517 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:33.524+0000 7f76d4a21040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-27T03:05:33.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.546+0000 7f54d2fc6640 10 monclient: get_monmap_and_config 2024-03-27T03:05:33.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.546+0000 7f54d2fc6640 10 monclient: build_initial_monmap 2024-03-27T03:05:33.539 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.546+0000 7f54d2fc6640 10 monclient: monmap: 2024-03-27T03:05:33.540 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:33.540 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.540 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:05:33.547850+0000 2024-03-27T03:05:33.540 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:05:33.547850+0000 2024-03-27T03:05:33.540 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:33.540 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.540 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:33.540 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:33.540 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:33.540 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.547+0000 7f54d2fc6640 1 Processor -- start 2024-03-27T03:05:33.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.547+0000 7f54d2fc6640 1 -- start start 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.547+0000 7f54d2fc6640 10 monclient: init 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54d2fc6640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54d2fc6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54d2fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f54cc09ac40 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54d2fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f54cc09c870 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54d2fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f54cc0b4360 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54d2fc6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54cc09dec0 con 0x7f54cc09ac40 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54d2fc6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54cc09da30 con 0x7f54cc09c870 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54d2fc6640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54cc0b39c0 con 0x7f54cc0b4360 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54d2fc6640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54d2fc6640 10 monclient(hunting): authenticate will time out at 814.866981s 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cbfff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f54cc09c870 0x7f54cc13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34734/0 (socket says 172.21.15.196:34734) 2024-03-27T03:05:33.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cbfff640 1 -- 172.21.15.196:0/2096028448 learned_addr learned my addr 172.21.15.196:0/2096028448 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863180718 0 0) 0x7f54cc09da30 con 0x7f54cc09c870 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient(hunting): my global_id is 4117 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54bc003520 con 0x7f54cc09c870 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462307624 0 0) 0x7f54cc0b39c0 con 0x7f54cc0b4360 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient(hunting): my global_id is 4130 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54cc09da30 con 0x7f54cc0b4360 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801046791 0 0) 0x7f54c0001110 con 0x7f54cc09ac40 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient(hunting): my global_id is 4116 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54cc0b39c0 con 0x7f54cc09ac40 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1879431645 0 0) 0x7f54bc003520 con 0x7f54cc09c870 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 >> v1:172.21.15.153:6790/0 conn(0x7f54cc0b4360 legacy=0x7f54cc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 >> v1:172.21.15.153:6789/0 conn(0x7f54cc09ac40 legacy=0x7f54cc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient: _finish_hunting 0 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 1 monclient: found mon.noname-c 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc0b5160 con 0x7f54cc09c870 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient: _finish_auth 0 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient: _check_auth_tickets 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54d2fc6640 5 monclient: authenticate success, global_id 4117 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f54b8001690 con 0x7f54cc09c870 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54cb7fe640 10 monclient: dump: 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.553 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.548+0000 7f54d2fc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54b8001870 con 0x7f54cc09c870 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54d2fc6640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54cb7fe640 1 -- 172.21.15.196:0/2096028448 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f54b8001c80 con 0x7f54cc09c870 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54d2fc6640 4 set_mon_vals no callback set 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54cb7fe640 10 monclient: dump: 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.554 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54d2fc6640 10 monclient: shutdown 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54d2fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54d2fc6640 1 -- 172.21.15.196:0/2096028448 >> v1:172.21.15.196:6789/0 conn(0x7f54cc09c870 legacy=0x7f54cc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54d2fc6640 1 -- 172.21.15.196:0/2096028448 shutdown_connections 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54d2fc6640 1 -- 172.21.15.196:0/2096028448 >> 172.21.15.196:0/2096028448 conn(0x7f54cc066b00 msgr2=0x7f54cc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54d2fc6640 1 -- 172.21.15.196:0/2096028448 shutdown_connections 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.549+0000 7f54d2fc6640 1 -- 172.21.15.196:0/2096028448 wait complete. 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.550+0000 7f54d2fc6640 10 monclient: build_initial_monmap 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.550+0000 7f54d2fc6640 10 monclient: monmap: 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:33.555 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.550+0000 7f54d2fc6640 1 librados: starting msgr at 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.550+0000 7f54d2fc6640 1 librados: starting objecter 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.550+0000 7f54d2fc6640 1 Processor -- start 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.550+0000 7f54d2fc6640 1 -- start start 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.550+0000 7f54d2fc6640 1 librados: setting wanted keys 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.550+0000 7f54d2fc6640 1 librados: calling monclient init 2024-03-27T03:05:33.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.550+0000 7f54d2fc6640 10 monclient: init 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54d2fc6640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54d2fc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54d2fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f54cc09ac40 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54d2fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f54cc1db230 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54d2fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f54cc0b0910 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54d2fc6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54cc1da690 con 0x7f54cc1db230 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54d2fc6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54cc0b1910 con 0x7f54cc09ac40 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54d2fc6640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54cc0acd00 con 0x7f54cc0b0910 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54d2fc6640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54d2fc6640 10 monclient(hunting): authenticate will time out at 814.870066s 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54d0d62640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f54cc09ac40 0x7f54cc1dab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34748/0 (socket says 172.21.15.196:34748) 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54d0d62640 1 -- 172.21.15.196:0/531209105 learned_addr learned my addr 172.21.15.196:0/531209105 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 client.?.objecter ms_handle_connect 0x7f54cc09ac40 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:33.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 client.?.objecter ms_handle_connect 0x7f54cc1db230 2024-03-27T03:05:33.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:33.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 client.?.objecter ms_handle_connect 0x7f54cc0b0910 2024-03-27T03:05:33.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:33.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107676694 0 0) 0x7f54cc0b1910 con 0x7f54cc09ac40 2024-03-27T03:05:33.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): my global_id is 4120 2024-03-27T03:05:33.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b0004eb0 con 0x7f54cc09ac40 2024-03-27T03:05:33.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195727878 0 0) 0x7f54cc0acd00 con 0x7f54cc0b0910 2024-03-27T03:05:33.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): my global_id is 4133 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54cc0b1910 con 0x7f54cc0b0910 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943478375 0 0) 0x7f54cc1da690 con 0x7f54cc1db230 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): my global_id is 4119 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54cc0acd00 con 0x7f54cc1db230 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4272130448 0 0) 0x7f54b0004eb0 con 0x7f54cc09ac40 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54cc1da690 con 0x7f54cc09ac40 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f54c0001860 con 0x7f54cc09ac40 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): dump: 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2425703153 0 0) 0x7f54cc0b1910 con 0x7f54cc0b0910 2024-03-27T03:05:33.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54b0004eb0 con 0x7f54cc0b0910 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f54c4001a20 con 0x7f54cc0b0910 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.551+0000 7f54c97fa640 10 monclient(hunting): dump: 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1797608194 0 0) 0x7f54cc0acd00 con 0x7f54cc1db230 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54cc0b1910 con 0x7f54cc1db230 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f54b8001b80 con 0x7f54cc1db230 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 10 monclient(hunting): dump: 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.562 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1653487171 0 0) 0x7f54cc1da690 con 0x7f54cc09ac40 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 >> v1:172.21.15.153:6790/0 conn(0x7f54cc0b0910 legacy=0x7f54cc1e1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 >> v1:172.21.15.153:6789/0 conn(0x7f54cc1db230 legacy=0x7f54cc0b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 10 monclient: _finish_hunting 0 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 1 monclient: found mon.b 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc0ace90 con 0x7f54cc09ac40 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 10 monclient: _finish_auth 0 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 10 monclient: _check_auth_tickets 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54d2fc6640 5 monclient: authenticate success, global_id 4120 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54d2fc6640 10 monclient: _renew_subs 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54d2fc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54d2fc6640 1 -- 172.21.15.196:0/531209105 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54cc0af5f0 con 0x7f54cc09ac40 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54d2fc6640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54d2fc6640 10 monclient: _renew_subs 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54d2fc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54d2fc6640 1 -- 172.21.15.196:0/531209105 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54cc1e3470 con 0x7f54cc09ac40 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54d2fc6640 1 librados: init done 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54c0001d50 con 0x7f54cc09ac40 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f54c0003610 con 0x7f54cc09ac40 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54caffd640 4 set_mon_vals no callback set 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 10 monclient: dump: 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.564 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f54c0003bc0 con 0x7f54cc09ac40 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3639171274 0 0) 0x7f54c0003e70 con 0x7f54cc09ac40 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 10 client.4120.objecter ms_dispatch 0x7f54cc000b90 osd_map(4..4 src has 1..4) v4 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 3 client.4120.objecter handle_osd_map got epochs [4,4] > 0 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 3 client.4120.objecter handle_osd_map decoding full epoch 4 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.552+0000 7f54c97fa640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.553+0000 7f54d2fc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.553+0000 7f54d2fc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:33.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.553+0000 7f54d2fc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.553+0000 7f54d2fc6640 1 -- 172.21.15.196:0/531209105 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54ac005080 con 0x7f54cc09ac40 2024-03-27T03:05:33.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.555+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f54c0005820 con 0x7f54cc09ac40 2024-03-27T03:05:33.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.555+0000 7f54c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:33.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.555+0000 7f54c97fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:33.567 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:33.575+0000 7f9854a42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-27T03:05:33.572 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:33.579+0000 7f76d4a21040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-27T03:05:33.616 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:33.624+0000 7f76d4a21040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-27T03:05:33.642 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:33.650+0000 7f9854a42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-27T03:05:33.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.663+0000 7f54d2fc6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2193eae0-0507-4a25-b242-eb5d61931b56", "id": 2}] 2024-03-27T03:05:33.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.663+0000 7f54d2fc6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2193eae0-0507-4a25-b242-eb5d61931b56", "id": 2}] 2024-03-27T03:05:33.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.663+0000 7f54d2fc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.663+0000 7f54d2fc6640 1 -- 172.21.15.196:0/531209105 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2193eae0-0507-4a25-b242-eb5d61931b56", "id": 2} v 0) v1 -- 0x7f54ac0058b0 con 0x7f54cc09ac40 2024-03-27T03:05:33.659 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.667+0000 7f54c97fa640 1 -- 172.21.15.196:0/531209105 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2193eae0-0507-4a25-b242-eb5d61931b56", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1984390197 0 216599125) 0x7f54c0004030 con 0x7f54cc09ac40 2024-03-27T03:05:33.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.667+0000 7f54c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2193eae0-0507-4a25-b242-eb5d61931b56", "id": 2}] 2024-03-27T03:05:33.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.667+0000 7f54c97fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:33.660 INFO:teuthology.orchestra.run.smithi196.stdout:2 2024-03-27T03:05:33.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.667+0000 7f54d2fc6640 10 librados: watch_flush enter 2024-03-27T03:05:33.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.667+0000 7f54d2fc6640 10 librados: watch_flush exit 2024-03-27T03:05:33.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.667+0000 7f54d2fc6640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-27T03:05:33.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.667+0000 7f54d2fc6640 10 client.4120.objecter successfully canceled tick 2024-03-27T03:05:33.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.668+0000 7f54d2fc6640 10 monclient: shutdown 2024-03-27T03:05:33.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.668+0000 7f54d2fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:33.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.668+0000 7f54d2fc6640 1 -- 172.21.15.196:0/531209105 >> v1:172.21.15.196:6789/0 conn(0x7f54cc09ac40 legacy=0x7f54cc1dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.668+0000 7f54d2fc6640 1 -- 172.21.15.196:0/531209105 shutdown_connections 2024-03-27T03:05:33.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.668+0000 7f54d2fc6640 1 -- 172.21.15.196:0/531209105 >> 172.21.15.196:0/531209105 conn(0x7f54cc066b00 msgr2=0x7f54cc0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:33.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.668+0000 7f54d2fc6640 1 -- 172.21.15.196:0/531209105 shutdown_connections 2024-03-27T03:05:33.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.668+0000 7f54d2fc6640 1 -- 172.21.15.196:0/531209105 wait complete. 2024-03-27T03:05:33.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.668+0000 7f54d2fc6640 1 librados: shutdown 2024-03-27T03:05:33.670 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:33.678+0000 7f76d4a21040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-27T03:05:33.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new a8bacc46-d45a-463a-9b8e-c4af076e5a30 3 2024-03-27T03:05:33.691 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:33.698+0000 7f9854a42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-27T03:05:33.712 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:33.720+0000 7f76d4a21040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-27T03:05:33.747 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:33.754+0000 7f9854a42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-27T03:05:33.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.777+0000 7f0a0833d640 10 monclient: get_monmap_and_config 2024-03-27T03:05:33.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.777+0000 7f0a0833d640 10 monclient: build_initial_monmap 2024-03-27T03:05:33.771 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.777+0000 7f0a0833d640 10 monclient: monmap: 2024-03-27T03:05:33.771 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:33.771 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.771 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:05:33.778904+0000 2024-03-27T03:05:33.771 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:05:33.778904+0000 2024-03-27T03:05:33.771 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:33.771 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.771 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:33.771 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:33.772 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:33.772 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.778+0000 7f0a0833d640 1 Processor -- start 2024-03-27T03:05:33.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.778+0000 7f0a0833d640 1 -- start start 2024-03-27T03:05:33.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.778+0000 7f0a0833d640 10 monclient: init 2024-03-27T03:05:33.781 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:33.781 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a0009ac40 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 10 monclient(hunting): picked mon.noname-b con 0x7f0a0009c870 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a000b4360 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0009dec0 con 0x7f0a0009ac40 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0009da30 con 0x7f0a000b4360 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000b39c0 con 0x7f0a0009c870 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 10 monclient(hunting): authenticate will time out at 815.097960s 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a068da640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f0a000b4360 0x7f0a0014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34752/0 (socket says 172.21.15.196:34752) 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a068da640 1 -- 172.21.15.196:0/102772821 learned_addr learned my addr 172.21.15.196:0/102772821 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124489146 0 0) 0x7f0a0009da30 con 0x7f0a000b4360 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient(hunting): my global_id is 4123 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09ec003520 con 0x7f0a000b4360 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180260346 0 0) 0x7f0a000b39c0 con 0x7f0a0009c870 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient(hunting): my global_id is 4136 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0009da30 con 0x7f0a0009c870 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573195457 0 0) 0x7f09fc001040 con 0x7f0a0009ac40 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient(hunting): my global_id is 4122 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a000b39c0 con 0x7f0a0009ac40 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1041327146 0 0) 0x7f09ec003520 con 0x7f0a000b4360 2024-03-27T03:05:33.783 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 >> v1:172.21.15.153:6790/0 conn(0x7f0a0009c870 legacy=0x7f0a0013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 >> v1:172.21.15.153:6789/0 conn(0x7f0a0009ac40 legacy=0x7f0a000b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient: _finish_hunting 0 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 monclient: found mon.noname-c 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000b5160 con 0x7f0a000b4360 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient: _finish_auth 0 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient: _check_auth_tickets 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f09f8001a10 con 0x7f0a000b4360 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 5 monclient: authenticate success, global_id 4123 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient: dump: 2024-03-27T03:05:33.784 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09f8001bf0 con 0x7f0a000b4360 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a0833d640 4 set_mon_vals no callback set 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 1 -- 172.21.15.196:0/102772821 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f09f8002000 con 0x7f0a000b4360 2024-03-27T03:05:33.785 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.779+0000 7f0a050d7640 10 monclient: dump: 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.780+0000 7f0a0833d640 10 monclient: shutdown 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.780+0000 7f0a0833d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.780+0000 7f0a0833d640 1 -- 172.21.15.196:0/102772821 >> v1:172.21.15.196:6789/0 conn(0x7f0a000b4360 legacy=0x7f0a0014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.786 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.780+0000 7f0a0833d640 1 -- 172.21.15.196:0/102772821 shutdown_connections 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.780+0000 7f0a0833d640 1 -- 172.21.15.196:0/102772821 >> 172.21.15.196:0/102772821 conn(0x7f0a00066b00 msgr2=0x7f0a00066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.780+0000 7f0a0833d640 1 -- 172.21.15.196:0/102772821 shutdown_connections 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.780+0000 7f0a0833d640 1 -- 172.21.15.196:0/102772821 wait complete. 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.781+0000 7f0a0833d640 10 monclient: build_initial_monmap 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.781+0000 7f0a0833d640 10 monclient: monmap: 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:33.787 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.781+0000 7f0a0833d640 1 librados: starting msgr at 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.781+0000 7f0a0833d640 1 librados: starting objecter 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.781+0000 7f0a0833d640 1 Processor -- start 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.781+0000 7f0a0833d640 1 -- start start 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.781+0000 7f0a0833d640 1 librados: setting wanted keys 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.781+0000 7f0a0833d640 1 librados: calling monclient init 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.781+0000 7f0a0833d640 10 monclient: init 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f0a0833d640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f0a0833d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f0a0833d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a0009ac40 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f0a0833d640 10 monclient(hunting): picked mon.noname-b con 0x7f0a000b0640 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f0a0833d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a000ad8a0 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f0a0833d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000b0ba0 con 0x7f0a000ad8a0 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f0a0833d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000b18b0 con 0x7f0a000b0640 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f0a0833d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a001401d0 con 0x7f0a0009ac40 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f0a0833d640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:33.788 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f0a0833d640 10 monclient(hunting): authenticate will time out at 815.101005s 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f0a058d8640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f0a000b0640 0x7f0a000ad190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34762/0 (socket says 172.21.15.196:34762) 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f0a058d8640 1 -- 172.21.15.196:0/4293359454 learned_addr learned my addr 172.21.15.196:0/4293359454 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 client.?.objecter ms_handle_connect 0x7f0a000b0640 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 client.?.objecter ms_handle_connect 0x7f0a0009ac40 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 client.?.objecter ms_handle_connect 0x7f0a000ad8a0 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187929088 0 0) 0x7f0a000b18b0 con 0x7f0a000b0640 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): my global_id is 4126 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e4004eb0 con 0x7f0a000b0640 2024-03-27T03:05:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227745661 0 0) 0x7f0a000b0ba0 con 0x7f0a000ad8a0 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): my global_id is 4125 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a000b18b0 con 0x7f0a000ad8a0 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895485698 0 0) 0x7f0a001401d0 con 0x7f0a0009ac40 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): my global_id is 4139 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a000b0ba0 con 0x7f0a0009ac40 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1077859366 0 0) 0x7f09e4004eb0 con 0x7f0a000b0640 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a001401d0 con 0x7f0a000b0640 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f09f00019b0 con 0x7f0a000b0640 2024-03-27T03:05:33.790 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): dump: 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 502100187 0 0) 0x7f0a000b18b0 con 0x7f0a000ad8a0 2024-03-27T03:05:33.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09e4004eb0 con 0x7f0a000ad8a0 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f09f8001f00 con 0x7f0a000ad8a0 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): dump: 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 601555018 0 0) 0x7f0a000b0ba0 con 0x7f0a0009ac40 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a000b18b0 con 0x7f0a0009ac40 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f09fc0017d0 con 0x7f0a0009ac40 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.782+0000 7f09f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 10 monclient(hunting): dump: 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1554288453 0 0) 0x7f0a001401d0 con 0x7f0a000b0640 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 >> v1:172.21.15.153:6790/0 conn(0x7f0a0009ac40 legacy=0x7f0a000afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 >> v1:172.21.15.153:6789/0 conn(0x7f0a000ad8a0 legacy=0x7f0a001e1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 1 monclient: found mon.b 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000b1030 con 0x7f0a000b0640 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 10 monclient: _finish_auth 0 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f0a0833d640 5 monclient: authenticate success, global_id 4126 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f0a0833d640 10 monclient: _renew_subs 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f0a0833d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f0a0833d640 1 -- 172.21.15.196:0/4293359454 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a001e3650 con 0x7f0a000b0640 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f0a0833d640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f0a0833d640 10 monclient: _renew_subs 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f0a0833d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f0a0833d640 1 -- 172.21.15.196:0/4293359454 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a001e39c0 con 0x7f0a000b0640 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f0a0833d640 1 librados: init done 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09f0003610 con 0x7f0a000b0640 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f09f0003920 con 0x7f0a000b0640 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f0a048d6640 4 set_mon_vals no callback set 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 10 monclient: dump: 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:33.795 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f09f0003e10 con 0x7f0a000b0640 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2689631685 0 0) 0x7f09f00057c0 con 0x7f0a000b0640 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 10 client.4126.objecter ms_dispatch 0x7f0a00000b90 osd_map(5..5 src has 1..5) v4 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 3 client.4126.objecter handle_osd_map got epochs [5,5] > 0 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 3 client.4126.objecter handle_osd_map decoding full epoch 5 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.783+0000 7f09f6ffd640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.784+0000 7f0a0833d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.784+0000 7f0a0833d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:33.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.784+0000 7f0a0833d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.784+0000 7f0a0833d640 1 -- 172.21.15.196:0/4293359454 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e8005080 con 0x7f0a000b0640 2024-03-27T03:05:33.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.785+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f09f0005a50 con 0x7f0a000b0640 2024-03-27T03:05:33.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.786+0000 7f09f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:33.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.786+0000 7f09f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:33.797 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:33.799+0000 7f9854a42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-27T03:05:33.829 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:33.836+0000 7f76d4a21040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-27T03:05:33.851 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.858+0000 7f0a0833d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a8bacc46-d45a-463a-9b8e-c4af076e5a30", "id": 3}] 2024-03-27T03:05:33.851 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.858+0000 7f0a0833d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a8bacc46-d45a-463a-9b8e-c4af076e5a30", "id": 3}] 2024-03-27T03:05:33.851 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.858+0000 7f0a0833d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:33.851 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.858+0000 7f0a0833d640 1 -- 172.21.15.196:0/4293359454 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a8bacc46-d45a-463a-9b8e-c4af076e5a30", "id": 3} v 0) v1 -- 0x7f09e80058b0 con 0x7f0a000b0640 2024-03-27T03:05:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.862+0000 7f09f6ffd640 1 -- 172.21.15.196:0/4293359454 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a8bacc46-d45a-463a-9b8e-c4af076e5a30", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3720199363 0 525046306) 0x7f09f00040c0 con 0x7f0a000b0640 2024-03-27T03:05:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.862+0000 7f09f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a8bacc46-d45a-463a-9b8e-c4af076e5a30", "id": 3}] 2024-03-27T03:05:33.855 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.862+0000 7f09f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:33.855 INFO:teuthology.orchestra.run.smithi196.stdout:3 2024-03-27T03:05:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.863+0000 7f0a0833d640 10 librados: watch_flush enter 2024-03-27T03:05:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.863+0000 7f0a0833d640 10 librados: watch_flush exit 2024-03-27T03:05:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.863+0000 7f0a0833d640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-27T03:05:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.863+0000 7f0a0833d640 10 client.4126.objecter successfully canceled tick 2024-03-27T03:05:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.863+0000 7f0a0833d640 10 monclient: shutdown 2024-03-27T03:05:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.863+0000 7f0a0833d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.863+0000 7f0a0833d640 1 -- 172.21.15.196:0/4293359454 >> v1:172.21.15.196:6789/0 conn(0x7f0a000b0640 legacy=0x7f0a000ad190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.863+0000 7f0a0833d640 1 -- 172.21.15.196:0/4293359454 shutdown_connections 2024-03-27T03:05:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.863+0000 7f0a0833d640 1 -- 172.21.15.196:0/4293359454 >> 172.21.15.196:0/4293359454 conn(0x7f0a00066b00 msgr2=0x7f0a0013a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.863+0000 7f0a0833d640 1 -- 172.21.15.196:0/4293359454 shutdown_connections 2024-03-27T03:05:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.863+0000 7f0a0833d640 1 -- 172.21.15.196:0/4293359454 wait complete. 2024-03-27T03:05:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:33.863+0000 7f0a0833d640 1 librados: shutdown 2024-03-27T03:05:33.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 125c9f30-749d-4e6a-8052-a3fcb44dfc43 4 2024-03-27T03:05:33.898 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:33.906+0000 7f76d4a21040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-27T03:05:33.911 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:33.919+0000 7f9854a42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-27T03:05:33.943 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:33.950+0000 7f76d4a21040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-27T03:05:33.985 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:33.992+0000 7f9854a42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-27T03:05:33.995 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:34.002+0000 7f76d4a21040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-27T03:05:34.006 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.013+0000 7f1cfaeba640 10 monclient: get_monmap_and_config 2024-03-27T03:05:34.006 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.013+0000 7f1cfaeba640 10 monclient: build_initial_monmap 2024-03-27T03:05:34.006 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.013+0000 7f1cfaeba640 10 monclient: monmap: 2024-03-27T03:05:34.006 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:34.007 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.007 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:05:34.014832+0000 2024-03-27T03:05:34.007 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:05:34.014832+0000 2024-03-27T03:05:34.007 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:34.007 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.007 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:34.007 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:34.007 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:34.007 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.016 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.014+0000 7f1cfaeba640 1 Processor -- start 2024-03-27T03:05:34.016 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.014+0000 7f1cfaeba640 1 -- start start 2024-03-27T03:05:34.016 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.014+0000 7f1cfaeba640 10 monclient: init 2024-03-27T03:05:34.016 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:34.016 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:05:34.016 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 10 monclient(hunting): picked mon.noname-c con 0x7f1cf409ac40 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:34.016 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 10 monclient(hunting): picked mon.noname-a con 0x7f1cf409c870 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:34.016 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 10 monclient(hunting): picked mon.noname-b con 0x7f1cf40b4360 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf409dec0 con 0x7f1cf409c870 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf409da30 con 0x7f1cf409ac40 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf40b39c0 con 0x7f1cf40b4360 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 10 monclient(hunting): authenticate will time out at 815.333900s 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf8c56640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f1cf409ac40 0x7f1cf40b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34766/0 (socket says 172.21.15.196:34766) 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf8c56640 1 -- 172.21.15.196:0/3886338582 learned_addr learned my addr 172.21.15.196:0/3886338582 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510989130 0 0) 0x7f1cf409da30 con 0x7f1cf409ac40 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient(hunting): my global_id is 4129 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce4003520 con 0x7f1cf409ac40 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021758276 0 0) 0x7f1cf40b39c0 con 0x7f1cf40b4360 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.017 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient(hunting): my global_id is 4142 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf409da30 con 0x7f1cf40b4360 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367465930 0 0) 0x7f1ce0001040 con 0x7f1cf409c870 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient(hunting): my global_id is 4128 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf40b39c0 con 0x7f1cf409c870 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4105063533 0 0) 0x7f1ce4003520 con 0x7f1cf409ac40 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 >> v1:172.21.15.153:6790/0 conn(0x7f1cf40b4360 legacy=0x7f1cf414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 >> v1:172.21.15.153:6789/0 conn(0x7f1cf409c870 legacy=0x7f1cf413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient: _finish_hunting 0 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 monclient: found mon.noname-c 2024-03-27T03:05:34.018 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf40b5160 con 0x7f1cf409ac40 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient: _finish_auth 0 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient: _check_auth_tickets 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f1ce8001690 con 0x7f1cf409ac40 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 5 monclient: authenticate success, global_id 4129 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient: dump: 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ce8001870 con 0x7f1cf409ac40 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 1 -- 172.21.15.196:0/3886338582 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f1ce8001c80 con 0x7f1cf409ac40 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cfaeba640 4 set_mon_vals no callback set 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.015+0000 7f1cf37fe640 10 monclient: dump: 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.016+0000 7f1cfaeba640 10 monclient: shutdown 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.016+0000 7f1cfaeba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.016+0000 7f1cfaeba640 1 -- 172.21.15.196:0/3886338582 >> v1:172.21.15.196:6789/0 conn(0x7f1cf409ac40 legacy=0x7f1cf40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.016+0000 7f1cfaeba640 1 -- 172.21.15.196:0/3886338582 shutdown_connections 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.016+0000 7f1cfaeba640 1 -- 172.21.15.196:0/3886338582 >> 172.21.15.196:0/3886338582 conn(0x7f1cf4066b00 msgr2=0x7f1cf4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.016+0000 7f1cfaeba640 1 -- 172.21.15.196:0/3886338582 shutdown_connections 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.016+0000 7f1cfaeba640 1 -- 172.21.15.196:0/3886338582 wait complete. 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.017+0000 7f1cfaeba640 10 monclient: build_initial_monmap 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.017+0000 7f1cfaeba640 10 monclient: monmap: 2024-03-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.017+0000 7f1cfaeba640 1 librados: starting msgr at 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.017+0000 7f1cfaeba640 1 librados: starting objecter 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.017+0000 7f1cfaeba640 1 Processor -- start 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.017+0000 7f1cfaeba640 1 -- start start 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.017+0000 7f1cfaeba640 1 librados: setting wanted keys 2024-03-27T03:05:34.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.017+0000 7f1cfaeba640 1 librados: calling monclient init 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.017+0000 7f1cfaeba640 10 monclient: init 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cfaeba640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cfaeba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cfaeba640 10 monclient(hunting): picked mon.noname-b con 0x7f1cf409ac40 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cfaeba640 10 monclient(hunting): picked mon.noname-a con 0x7f1cf40acd70 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cfaeba640 10 monclient(hunting): picked mon.noname-c con 0x7f1cf4139820 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cfaeba640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf40b1a60 con 0x7f1cf40acd70 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cfaeba640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf41d73f0 con 0x7f1cf409ac40 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cfaeba640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf41d8020 con 0x7f1cf4139820 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cfaeba640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cfaeba640 10 monclient(hunting): authenticate will time out at 815.336964s 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf8c56640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f1cf409ac40 0x7f1cf41d7910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34770/0 (socket says 172.21.15.196:34770) 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf8c56640 1 -- 172.21.15.196:0/2837713684 learned_addr learned my addr 172.21.15.196:0/2837713684 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 client.?.objecter ms_handle_connect 0x7f1cf409ac40 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 client.?.objecter ms_handle_connect 0x7f1cf4139820 2024-03-27T03:05:34.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 client.?.objecter ms_handle_connect 0x7f1cf40acd70 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343442634 0 0) 0x7f1cf41d73f0 con 0x7f1cf409ac40 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): my global_id is 4132 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd8004eb0 con 0x7f1cf409ac40 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225596561 0 0) 0x7f1cf41d8020 con 0x7f1cf4139820 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): my global_id is 4145 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf41d73f0 con 0x7f1cf4139820 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877589531 0 0) 0x7f1cf40b1a60 con 0x7f1cf40acd70 2024-03-27T03:05:34.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): my global_id is 4131 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf41d8020 con 0x7f1cf40acd70 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 370400868 0 0) 0x7f1cd8004eb0 con 0x7f1cf409ac40 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cf40b1a60 con 0x7f1cf409ac40 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f1ce8001bc0 con 0x7f1cf409ac40 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): dump: 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1423548456 0 0) 0x7f1cf41d73f0 con 0x7f1cf4139820 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cd8004eb0 con 0x7f1cf4139820 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f1cec001b50 con 0x7f1cf4139820 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 10 monclient(hunting): dump: 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.026 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.018+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 766583960 0 0) 0x7f1cf41d8020 con 0x7f1cf40acd70 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cf41d73f0 con 0x7f1cf40acd70 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f1ce0001690 con 0x7f1cf40acd70 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 10 monclient(hunting): dump: 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.027 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 944173956 0 0) 0x7f1cf40b1a60 con 0x7f1cf409ac40 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 >> v1:172.21.15.153:6790/0 conn(0x7f1cf4139820 legacy=0x7f1cf40b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 >> v1:172.21.15.153:6789/0 conn(0x7f1cf40acd70 legacy=0x7f1cf4139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 10 monclient: _finish_hunting 0 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 1 monclient: found mon.b 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf41d81b0 con 0x7f1cf409ac40 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 10 monclient: _finish_auth 0 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 10 monclient: _check_auth_tickets 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cfaeba640 5 monclient: authenticate success, global_id 4132 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cfaeba640 10 monclient: _renew_subs 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cfaeba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cfaeba640 1 -- 172.21.15.196:0/2837713684 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf40b07b0 con 0x7f1cf409ac40 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cfaeba640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cfaeba640 10 monclient: _renew_subs 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cfaeba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cfaeba640 1 -- 172.21.15.196:0/2837713684 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf40b0c90 con 0x7f1cf409ac40 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cfaeba640 1 librados: init done 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ce8004bf0 con 0x7f1cf409ac40 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f1ce8004f00 con 0x7f1cf409ac40 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf2ffd640 4 set_mon_vals no callback set 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 10 monclient: dump: 2024-03-27T03:05:34.029 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1ce80053f0 con 0x7f1cf409ac40 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 734343692 0 0) 0x7f1ce80069a0 con 0x7f1cf409ac40 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 10 client.4132.objecter ms_dispatch 0x7f1cf4000b90 osd_map(6..6 src has 1..6) v4 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 3 client.4132.objecter handle_osd_map got epochs [6,6] > 0 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 3 client.4132.objecter handle_osd_map decoding full epoch 6 2024-03-27T03:05:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.019+0000 7f1cf17fa640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-27T03:05:34.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.020+0000 7f1cfaeba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.020+0000 7f1cfaeba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.020+0000 7f1cfaeba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.020+0000 7f1cfaeba640 1 -- 172.21.15.196:0/2837713684 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd4005080 con 0x7f1cf409ac40 2024-03-27T03:05:34.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.021+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1ce8006540 con 0x7f1cf409ac40 2024-03-27T03:05:34.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.021+0000 7f1cf17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.021+0000 7f1cf17fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:34.034 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:34.041+0000 7f9854a42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-27T03:05:34.042 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:34.050+0000 7f76d4a21040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-27T03:05:34.086 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.093+0000 7f1cfaeba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "125c9f30-749d-4e6a-8052-a3fcb44dfc43", "id": 4}] 2024-03-27T03:05:34.086 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.093+0000 7f1cfaeba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "125c9f30-749d-4e6a-8052-a3fcb44dfc43", "id": 4}] 2024-03-27T03:05:34.086 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.093+0000 7f1cfaeba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.086 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.093+0000 7f1cfaeba640 1 -- 172.21.15.196:0/2837713684 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "125c9f30-749d-4e6a-8052-a3fcb44dfc43", "id": 4} v 0) v1 -- 0x7f1cd40058b0 con 0x7f1cf409ac40 2024-03-27T03:05:34.088 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:34.095+0000 7f9854a42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-27T03:05:34.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.098+0000 7f1cf17fa640 1 -- 172.21.15.196:0/2837713684 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "125c9f30-749d-4e6a-8052-a3fcb44dfc43", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2758710046 0 1697012583) 0x7f1ce8006b60 con 0x7f1cf409ac40 2024-03-27T03:05:34.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.098+0000 7f1cf17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "125c9f30-749d-4e6a-8052-a3fcb44dfc43", "id": 4}] 2024-03-27T03:05:34.091 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.098+0000 7f1cf17fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:34.091 INFO:teuthology.orchestra.run.smithi196.stdout:4 2024-03-27T03:05:34.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.098+0000 7f1cfaeba640 10 librados: watch_flush enter 2024-03-27T03:05:34.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.098+0000 7f1cfaeba640 10 librados: watch_flush exit 2024-03-27T03:05:34.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.098+0000 7f1cfaeba640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-27T03:05:34.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.098+0000 7f1cfaeba640 10 client.4132.objecter successfully canceled tick 2024-03-27T03:05:34.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.099+0000 7f1cfaeba640 10 monclient: shutdown 2024-03-27T03:05:34.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.099+0000 7f1cfaeba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:34.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.099+0000 7f1cfaeba640 1 -- 172.21.15.196:0/2837713684 >> v1:172.21.15.196:6789/0 conn(0x7f1cf409ac40 legacy=0x7f1cf41d7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.099+0000 7f1cfaeba640 1 -- 172.21.15.196:0/2837713684 shutdown_connections 2024-03-27T03:05:34.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.099+0000 7f1cfaeba640 1 -- 172.21.15.196:0/2837713684 >> 172.21.15.196:0/2837713684 conn(0x7f1cf4066b00 msgr2=0x7f1cf40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:34.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.099+0000 7f1cfaeba640 1 -- 172.21.15.196:0/2837713684 shutdown_connections 2024-03-27T03:05:34.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.099+0000 7f1cfaeba640 1 -- 172.21.15.196:0/2837713684 wait complete. 2024-03-27T03:05:34.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.099+0000 7f1cfaeba640 1 librados: shutdown 2024-03-27T03:05:34.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new c3395381-57c8-4362-9fe5-6d60312ce660 5 2024-03-27T03:05:34.137 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:34.145+0000 7f9854a42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-27T03:05:34.237 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.244+0000 7ff60cb9d640 10 monclient: get_monmap_and_config 2024-03-27T03:05:34.237 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.244+0000 7ff60cb9d640 10 monclient: build_initial_monmap 2024-03-27T03:05:34.238 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.244+0000 7ff60cb9d640 10 monclient: monmap: 2024-03-27T03:05:34.238 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:34.238 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.238 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:05:34.245871+0000 2024-03-27T03:05:34.238 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:05:34.245871+0000 2024-03-27T03:05:34.238 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:34.238 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.238 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:34.238 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:34.238 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:34.238 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.239 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.245+0000 7ff60cb9d640 1 Processor -- start 2024-03-27T03:05:34.239 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.245+0000 7ff60cb9d640 1 -- start start 2024-03-27T03:05:34.239 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.245+0000 7ff60cb9d640 10 monclient: init 2024-03-27T03:05:34.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:34.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:05:34.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff60809ac40 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:34.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff60809c870 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:34.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff6080b4360 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:34.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60809dec0 con 0x7ff60809ac40 2024-03-27T03:05:34.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60809da30 con 0x7ff6080b4360 2024-03-27T03:05:34.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6080b39c0 con 0x7ff60809c870 2024-03-27T03:05:34.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:34.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 10 monclient(hunting): authenticate will time out at 815.564933s 2024-03-27T03:05:34.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff606d9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ff6080b4360 0x7ff60814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34772/0 (socket says 172.21.15.196:34772) 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff606d9d640 1 -- 172.21.15.196:0/4266390664 learned_addr learned my addr 172.21.15.196:0/4266390664 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541595212 0 0) 0x7ff60809da30 con 0x7ff6080b4360 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient(hunting): my global_id is 4135 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f4003520 con 0x7ff6080b4360 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152085042 0 0) 0x7ff5f8001110 con 0x7ff60809ac40 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient(hunting): my global_id is 4134 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff60809da30 con 0x7ff60809ac40 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143816922 0 0) 0x7ff6080b39c0 con 0x7ff60809c870 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient(hunting): my global_id is 4148 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f8001110 con 0x7ff60809c870 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2676400395 0 0) 0x7ff5f4003520 con 0x7ff6080b4360 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 >> v1:172.21.15.153:6790/0 conn(0x7ff60809c870 legacy=0x7ff60813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 >> v1:172.21.15.153:6789/0 conn(0x7ff60809ac40 legacy=0x7ff6080b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient: _finish_hunting 0 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 monclient: found mon.noname-c 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6080b5160 con 0x7ff6080b4360 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient: _finish_auth 0 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient: _check_auth_tickets 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:34.250 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 5 monclient: authenticate success, global_id 4135 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff600001a10 con 0x7ff6080b4360 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient: dump: 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:34.251 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff600001bf0 con 0x7ff6080b4360 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 1 -- 172.21.15.196:0/4266390664 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff600002000 con 0x7ff6080b4360 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff60cb9d640 4 set_mon_vals no callback set 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.246+0000 7ff605d9b640 10 monclient: dump: 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.252 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.247+0000 7ff60cb9d640 10 monclient: shutdown 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.247+0000 7ff60cb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.247+0000 7ff60cb9d640 1 -- 172.21.15.196:0/4266390664 >> v1:172.21.15.196:6789/0 conn(0x7ff6080b4360 legacy=0x7ff60814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.247+0000 7ff60cb9d640 1 -- 172.21.15.196:0/4266390664 shutdown_connections 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.247+0000 7ff60cb9d640 1 -- 172.21.15.196:0/4266390664 >> 172.21.15.196:0/4266390664 conn(0x7ff608066b00 msgr2=0x7ff608066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.247+0000 7ff60cb9d640 1 -- 172.21.15.196:0/4266390664 shutdown_connections 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.247+0000 7ff60cb9d640 1 -- 172.21.15.196:0/4266390664 wait complete. 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.248+0000 7ff60cb9d640 10 monclient: build_initial_monmap 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.248+0000 7ff60cb9d640 10 monclient: monmap: 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.248+0000 7ff60cb9d640 1 librados: starting msgr at 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.248+0000 7ff60cb9d640 1 librados: starting objecter 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.248+0000 7ff60cb9d640 1 Processor -- start 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.248+0000 7ff60cb9d640 1 -- start start 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.248+0000 7ff60cb9d640 1 librados: setting wanted keys 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.248+0000 7ff60cb9d640 1 librados: calling monclient init 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.248+0000 7ff60cb9d640 10 monclient: init 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff60809ac40 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:34.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff6080acfa0 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff6080b0880 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6081d7550 con 0x7ff6080acfa0 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6081d80f0 con 0x7ff60809ac40 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6081d71e0 con 0x7ff6080b0880 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 10 monclient(hunting): authenticate will time out at 815.567917s 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60659c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ff60809ac40 0x7ff6080afab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34778/0 (socket says 172.21.15.196:34778) 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60659c640 1 -- 172.21.15.196:0/1112852620 learned_addr learned my addr 172.21.15.196:0/1112852620 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 client.?.objecter ms_handle_connect 0x7ff60809ac40 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 client.?.objecter ms_handle_connect 0x7ff6080acfa0 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480454892 0 0) 0x7ff6081d80f0 con 0x7ff60809ac40 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): my global_id is 4138 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e8004eb0 con 0x7ff60809ac40 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 client.?.objecter ms_handle_connect 0x7ff6080b0880 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898393156 0 0) 0x7ff6081d7550 con 0x7ff6080acfa0 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): my global_id is 4137 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6081d80f0 con 0x7ff6080acfa0 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615645736 0 0) 0x7ff6081d71e0 con 0x7ff6080b0880 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): my global_id is 4151 2024-03-27T03:05:34.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6081d7550 con 0x7ff6080b0880 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 409853564 0 0) 0x7ff5e8004eb0 con 0x7ff60809ac40 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6081d71e0 con 0x7ff60809ac40 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff5f8001920 con 0x7ff60809ac40 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): dump: 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.257 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14419132 0 0) 0x7ff6081d80f0 con 0x7ff6080acfa0 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5e8004eb0 con 0x7ff6080acfa0 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff5f00018b0 con 0x7ff6080acfa0 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient(hunting): dump: 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.258 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2626682030 0 0) 0x7ff6081d71e0 con 0x7ff60809ac40 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 >> v1:172.21.15.153:6790/0 conn(0x7ff6080b0880 legacy=0x7ff608145ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 >> v1:172.21.15.153:6789/0 conn(0x7ff6080acfa0 legacy=0x7ff6081d79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient: _finish_hunting 0 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 monclient: found mon.b 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6080b0cd0 con 0x7ff60809ac40 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient: _finish_auth 0 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 10 monclient: _check_auth_tickets 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff5feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 5 monclient: authenticate success, global_id 4138 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 10 monclient: _renew_subs 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 1 -- 172.21.15.196:0/1112852620 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6081461f0 con 0x7ff60809ac40 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 10 monclient: _renew_subs 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 1 -- 172.21.15.196:0/1112852620 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6081e3b10 con 0x7ff60809ac40 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.249+0000 7ff60cb9d640 1 librados: init done 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5f8003610 con 0x7ff60809ac40 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff5feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff5f80039e0 con 0x7ff60809ac40 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff5feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff5feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff5feffd640 10 monclient: dump: 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff60559a640 4 set_mon_vals no callback set 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff5f8003f90 con 0x7ff60809ac40 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3528040947 0 0) 0x7ff5f8006770 con 0x7ff60809ac40 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff5feffd640 10 client.4138.objecter ms_dispatch 0x7ff608000b90 osd_map(7..7 src has 1..7) v4 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff5feffd640 3 client.4138.objecter handle_osd_map got epochs [7,7] > 0 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff5feffd640 3 client.4138.objecter handle_osd_map decoding full epoch 7 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff5feffd640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff60cb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.250+0000 7ff60cb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.251+0000 7ff60cb9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.251+0000 7ff60cb9d640 1 -- 172.21.15.196:0/1112852620 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ec005080 con 0x7ff60809ac40 2024-03-27T03:05:34.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.252+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff5f80043d0 con 0x7ff60809ac40 2024-03-27T03:05:34.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.252+0000 7ff5feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.252+0000 7ff5feffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:34.317 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.324+0000 7ff60cb9d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c3395381-57c8-4362-9fe5-6d60312ce660", "id": 5}] 2024-03-27T03:05:34.317 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.324+0000 7ff60cb9d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c3395381-57c8-4362-9fe5-6d60312ce660", "id": 5}] 2024-03-27T03:05:34.317 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.324+0000 7ff60cb9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.317 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.324+0000 7ff60cb9d640 1 -- 172.21.15.196:0/1112852620 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c3395381-57c8-4362-9fe5-6d60312ce660", "id": 5} v 0) v1 -- 0x7ff5ec0058b0 con 0x7ff60809ac40 2024-03-27T03:05:34.321 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.328+0000 7ff5feffd640 1 -- 172.21.15.196:0/1112852620 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c3395381-57c8-4362-9fe5-6d60312ce660", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 605053156 0 1988412176) 0x7ff5f8006130 con 0x7ff60809ac40 2024-03-27T03:05:34.321 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.328+0000 7ff5feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c3395381-57c8-4362-9fe5-6d60312ce660", "id": 5}] 2024-03-27T03:05:34.321 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.328+0000 7ff5feffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:34.321 INFO:teuthology.orchestra.run.smithi196.stdout:5 2024-03-27T03:05:34.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.329+0000 7ff60cb9d640 10 librados: watch_flush enter 2024-03-27T03:05:34.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.329+0000 7ff60cb9d640 10 librados: watch_flush exit 2024-03-27T03:05:34.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.329+0000 7ff60cb9d640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-27T03:05:34.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.329+0000 7ff60cb9d640 10 client.4138.objecter successfully canceled tick 2024-03-27T03:05:34.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.329+0000 7ff60cb9d640 10 monclient: shutdown 2024-03-27T03:05:34.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.329+0000 7ff60cb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:34.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.329+0000 7ff60cb9d640 1 -- 172.21.15.196:0/1112852620 >> v1:172.21.15.196:6789/0 conn(0x7ff60809ac40 legacy=0x7ff6080afab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.329+0000 7ff60cb9d640 1 -- 172.21.15.196:0/1112852620 shutdown_connections 2024-03-27T03:05:34.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.329+0000 7ff60cb9d640 1 -- 172.21.15.196:0/1112852620 >> 172.21.15.196:0/1112852620 conn(0x7ff608066b00 msgr2=0x7ff60813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:34.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.329+0000 7ff60cb9d640 1 -- 172.21.15.196:0/1112852620 shutdown_connections 2024-03-27T03:05:34.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.329+0000 7ff60cb9d640 1 -- 172.21.15.196:0/1112852620 wait complete. 2024-03-27T03:05:34.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.329+0000 7ff60cb9d640 1 librados: shutdown 2024-03-27T03:05:34.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new ddcca93d-aa6c-4c53-b3f3-7f49eef32854 6 2024-03-27T03:05:34.422 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:34.429+0000 7f76d4a21040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-27T03:05:34.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.439+0000 7fdc70e3e640 10 monclient: get_monmap_and_config 2024-03-27T03:05:34.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.439+0000 7fdc70e3e640 10 monclient: build_initial_monmap 2024-03-27T03:05:34.433 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.439+0000 7fdc70e3e640 10 monclient: monmap: 2024-03-27T03:05:34.433 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:34.433 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.433 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:05:34.440838+0000 2024-03-27T03:05:34.433 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:05:34.440838+0000 2024-03-27T03:05:34.433 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:34.433 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.433 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:34.433 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:34.433 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:34.433 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.440+0000 7fdc70e3e640 1 Processor -- start 2024-03-27T03:05:34.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.440+0000 7fdc70e3e640 1 -- start start 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.440+0000 7fdc70e3e640 10 monclient: init 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 10 monclient(hunting): picked mon.noname-c con 0x7fdc6c09ac40 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 10 monclient(hunting): picked mon.noname-b con 0x7fdc6c09c870 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 10 monclient(hunting): picked mon.noname-a con 0x7fdc6c0b4360 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c09dec0 con 0x7fdc6c0b4360 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c09da30 con 0x7fdc6c09ac40 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c0b39c0 con 0x7fdc6c09c870 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 10 monclient(hunting): authenticate will time out at 815.759927s 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc6a59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdc6c09ac40 0x7fdc6c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34786/0 (socket says 172.21.15.196:34786) 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc6a59c640 1 -- 172.21.15.196:0/20152959 learned_addr learned my addr 172.21.15.196:0/20152959 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626086637 0 0) 0x7fdc6c09da30 con 0x7fdc6c09ac40 2024-03-27T03:05:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient(hunting): my global_id is 4141 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc58003520 con 0x7fdc6c09ac40 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960168930 0 0) 0x7fdc64001490 con 0x7fdc6c0b4360 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient(hunting): my global_id is 4140 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc6c09da30 con 0x7fdc6c0b4360 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453097777 0 0) 0x7fdc6c0b39c0 con 0x7fdc6c09c870 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient(hunting): my global_id is 4154 2024-03-27T03:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc64001490 con 0x7fdc6c09c870 2024-03-27T03:05:34.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3327605143 0 0) 0x7fdc58003520 con 0x7fdc6c09ac40 2024-03-27T03:05:34.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 >> v1:172.21.15.153:6790/0 conn(0x7fdc6c09c870 legacy=0x7fdc6c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 >> v1:172.21.15.153:6789/0 conn(0x7fdc6c0b4360 legacy=0x7fdc6c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient: _finish_hunting 0 2024-03-27T03:05:34.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 monclient: found mon.noname-c 2024-03-27T03:05:34.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:34.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c0b5160 con 0x7fdc6c09ac40 2024-03-27T03:05:34.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient: _finish_auth 0 2024-03-27T03:05:34.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient: _check_auth_tickets 2024-03-27T03:05:34.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdc5c001690 con 0x7fdc6c09ac40 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 5 monclient: authenticate success, global_id 4141 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient: dump: 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.446 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc5c001870 con 0x7fdc6c09ac40 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 1 -- 172.21.15.196:0/20152959 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdc5c001c80 con 0x7fdc6c09ac40 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc70e3e640 4 set_mon_vals no callback set 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.441+0000 7fdc69d9b640 10 monclient: dump: 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.447 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.442+0000 7fdc70e3e640 10 monclient: shutdown 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.442+0000 7fdc70e3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.442+0000 7fdc70e3e640 1 -- 172.21.15.196:0/20152959 >> v1:172.21.15.196:6789/0 conn(0x7fdc6c09ac40 legacy=0x7fdc6c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.442+0000 7fdc70e3e640 1 -- 172.21.15.196:0/20152959 shutdown_connections 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.442+0000 7fdc70e3e640 1 -- 172.21.15.196:0/20152959 >> 172.21.15.196:0/20152959 conn(0x7fdc6c066b00 msgr2=0x7fdc6c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.442+0000 7fdc70e3e640 1 -- 172.21.15.196:0/20152959 shutdown_connections 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.442+0000 7fdc70e3e640 1 -- 172.21.15.196:0/20152959 wait complete. 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.443+0000 7fdc70e3e640 10 monclient: build_initial_monmap 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.443+0000 7fdc70e3e640 10 monclient: monmap: 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:34.448 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.443+0000 7fdc70e3e640 1 librados: starting msgr at 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.443+0000 7fdc70e3e640 1 librados: starting objecter 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.443+0000 7fdc70e3e640 1 Processor -- start 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.443+0000 7fdc70e3e640 1 -- start start 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.443+0000 7fdc70e3e640 1 librados: setting wanted keys 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.443+0000 7fdc70e3e640 1 librados: calling monclient init 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.443+0000 7fdc70e3e640 10 monclient: init 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc70e3e640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc70e3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:05:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc70e3e640 10 monclient(hunting): picked mon.noname-a con 0x7fdc6c09ac40 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:34.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc70e3e640 10 monclient(hunting): picked mon.noname-b con 0x7fdc6c0b5e30 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:34.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc70e3e640 10 monclient(hunting): picked mon.noname-c con 0x7fdc6c141ec0 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:34.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc70e3e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c0acfc0 con 0x7fdc6c09ac40 2024-03-27T03:05:34.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc70e3e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c1d72c0 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc70e3e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c140cd0 con 0x7fdc6c141ec0 2024-03-27T03:05:34.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc70e3e640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:34.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc70e3e640 10 monclient(hunting): authenticate will time out at 815.763000s 2024-03-27T03:05:34.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc63fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdc6c0b5e30 0x7fdc6c1417b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34788/0 (socket says 172.21.15.196:34788) 2024-03-27T03:05:34.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc63fff640 1 -- 172.21.15.196:0/1073535206 learned_addr learned my addr 172.21.15.196:0/1073535206 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:34.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 client.?.objecter ms_handle_connect 0x7fdc6c0b5e30 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 client.?.objecter ms_handle_connect 0x7fdc6c09ac40 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 client.?.objecter ms_handle_connect 0x7fdc6c141ec0 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762917302 0 0) 0x7fdc6c1d72c0 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): my global_id is 4144 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc4c004eb0 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124746007 0 0) 0x7fdc6c0acfc0 con 0x7fdc6c09ac40 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): my global_id is 4143 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc6c1d72c0 con 0x7fdc6c09ac40 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604188294 0 0) 0x7fdc6c140cd0 con 0x7fdc6c141ec0 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): my global_id is 4157 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc6c0acfc0 con 0x7fdc6c141ec0 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1370943164 0 0) 0x7fdc4c004eb0 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc6c140cd0 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdc54001a10 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): dump: 2024-03-27T03:05:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3941393509 0 0) 0x7fdc6c1d72c0 con 0x7fdc6c09ac40 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc4c004eb0 con 0x7fdc6c09ac40 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdc5c001b80 con 0x7fdc6c09ac40 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:05:34.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): dump: 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 346444769 0 0) 0x7fdc6c0acfc0 con 0x7fdc6c141ec0 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc6c1d72c0 con 0x7fdc6c141ec0 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdc64001a40 con 0x7fdc6c141ec0 2024-03-27T03:05:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient(hunting): dump: 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3568992826 0 0) 0x7fdc6c140cd0 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 >> v1:172.21.15.153:6790/0 conn(0x7fdc6c141ec0 legacy=0x7fdc6c1e1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 >> v1:172.21.15.153:6789/0 conn(0x7fdc6c09ac40 legacy=0x7fdc6c1d77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 monclient: found mon.b 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.444+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c1d7ef0 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 10 monclient: _finish_auth 0 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 5 monclient: authenticate success, global_id 4144 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 10 monclient: _renew_subs 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 1 -- 172.21.15.196:0/1073535206 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c1e3280 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 10 monclient: _renew_subs 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 1 -- 172.21.15.196:0/1073535206 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c1e35f0 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 1 librados: init done 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc54001cf0 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdc54003a60 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 10 monclient: dump: 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.457 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc6959a640 4 set_mon_vals no callback set 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdc54003fd0 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1132653096 0 0) 0x7fdc54006980 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 10 client.4144.objecter ms_dispatch 0x7fdc6c000b90 osd_map(8..8 src has 1..8) v4 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 3 client.4144.objecter handle_osd_map got epochs [8,8] > 0 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 3 client.4144.objecter handle_osd_map decoding full epoch 8 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc62ffd640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.445+0000 7fdc70e3e640 1 -- 172.21.15.196:0/1073535206 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc50005080 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.447+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdc54004460 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.447+0000 7fdc62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.447+0000 7fdc62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:34.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.521+0000 7fdc70e3e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ddcca93d-aa6c-4c53-b3f3-7f49eef32854", "id": 6}] 2024-03-27T03:05:34.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.521+0000 7fdc70e3e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ddcca93d-aa6c-4c53-b3f3-7f49eef32854", "id": 6}] 2024-03-27T03:05:34.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.521+0000 7fdc70e3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.514 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.521+0000 7fdc70e3e640 1 -- 172.21.15.196:0/1073535206 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ddcca93d-aa6c-4c53-b3f3-7f49eef32854", "id": 6} v 0) v1 -- 0x7fdc500058b0 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.517 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.525+0000 7fdc62ffd640 1 -- 172.21.15.196:0/1073535206 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ddcca93d-aa6c-4c53-b3f3-7f49eef32854", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3603018116 0 1113811849) 0x7fdc54006460 con 0x7fdc6c0b5e30 2024-03-27T03:05:34.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.525+0000 7fdc62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ddcca93d-aa6c-4c53-b3f3-7f49eef32854", "id": 6}] 2024-03-27T03:05:34.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.525+0000 7fdc62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:34.518 INFO:teuthology.orchestra.run.smithi196.stdout:6 2024-03-27T03:05:34.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.525+0000 7fdc70e3e640 10 librados: watch_flush enter 2024-03-27T03:05:34.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.525+0000 7fdc70e3e640 10 librados: watch_flush exit 2024-03-27T03:05:34.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.525+0000 7fdc70e3e640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-27T03:05:34.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.525+0000 7fdc70e3e640 10 client.4144.objecter successfully canceled tick 2024-03-27T03:05:34.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.526+0000 7fdc70e3e640 10 monclient: shutdown 2024-03-27T03:05:34.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.526+0000 7fdc70e3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:34.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.526+0000 7fdc70e3e640 1 -- 172.21.15.196:0/1073535206 >> v1:172.21.15.196:6789/0 conn(0x7fdc6c0b5e30 legacy=0x7fdc6c1417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.526+0000 7fdc70e3e640 1 -- 172.21.15.196:0/1073535206 shutdown_connections 2024-03-27T03:05:34.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.526+0000 7fdc70e3e640 1 -- 172.21.15.196:0/1073535206 >> 172.21.15.196:0/1073535206 conn(0x7fdc6c066b00 msgr2=0x7fdc6c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:34.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.526+0000 7fdc70e3e640 1 -- 172.21.15.196:0/1073535206 shutdown_connections 2024-03-27T03:05:34.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.526+0000 7fdc70e3e640 1 -- 172.21.15.196:0/1073535206 wait complete. 2024-03-27T03:05:34.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.526+0000 7fdc70e3e640 1 librados: shutdown 2024-03-27T03:05:34.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 47e2cc28-454d-43f2-b42c-d3b267a87d2c 7 2024-03-27T03:05:34.542 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:34.550+0000 7f9854a42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-27T03:05:34.558 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:34.566+0000 7f76d4a21040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-27T03:05:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.635+0000 7fdf7c206640 10 monclient: get_monmap_and_config 2024-03-27T03:05:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.635+0000 7fdf7c206640 10 monclient: build_initial_monmap 2024-03-27T03:05:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.635+0000 7fdf7c206640 10 monclient: monmap: 2024-03-27T03:05:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:05:34.636433+0000 2024-03-27T03:05:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:05:34.636433+0000 2024-03-27T03:05:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:34.629 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.635+0000 7fdf7c206640 1 Processor -- start 2024-03-27T03:05:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf7c206640 1 -- start start 2024-03-27T03:05:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf7c206640 10 monclient: init 2024-03-27T03:05:34.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf7c206640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:34.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf7c206640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:05:34.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf7c206640 10 monclient(hunting): picked mon.noname-b con 0x7fdf7409ac40 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:34.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf7c206640 10 monclient(hunting): picked mon.noname-c con 0x7fdf7409c870 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:34.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf7c206640 10 monclient(hunting): picked mon.noname-a con 0x7fdf740b4360 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:34.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf7c206640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7409dec0 con 0x7fdf740b4360 2024-03-27T03:05:34.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf7c206640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7409da30 con 0x7fdf7409c870 2024-03-27T03:05:34.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf7c206640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740b39c0 con 0x7fdf7409ac40 2024-03-27T03:05:34.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf7c206640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:34.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf7c206640 10 monclient(hunting): authenticate will time out at 815.955503s 2024-03-27T03:05:34.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf797a1640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdf7409c870 0x7fdf7413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34792/0 (socket says 172.21.15.196:34792) 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.636+0000 7fdf797a1640 1 -- 172.21.15.196:0/3640697731 learned_addr learned my addr 172.21.15.196:0/3640697731 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727205308 0 0) 0x7fdf7409da30 con 0x7fdf7409c870 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient(hunting): my global_id is 4147 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf60003520 con 0x7fdf7409c870 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712819056 0 0) 0x7fdf6c001490 con 0x7fdf740b4360 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient(hunting): my global_id is 4146 2024-03-27T03:05:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf7409da30 con 0x7fdf740b4360 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013536230 0 0) 0x7fdf740b39c0 con 0x7fdf7409ac40 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient(hunting): my global_id is 4160 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf6c001490 con 0x7fdf7409ac40 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2475260771 0 0) 0x7fdf60003520 con 0x7fdf7409c870 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 >> v1:172.21.15.153:6790/0 conn(0x7fdf7409ac40 legacy=0x7fdf740b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 >> v1:172.21.15.153:6789/0 conn(0x7fdf740b4360 legacy=0x7fdf7414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient: _finish_hunting 0 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 monclient: found mon.noname-c 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf740b5160 con 0x7fdf7409c870 2024-03-27T03:05:34.643 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient: _finish_auth 0 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient: _check_auth_tickets 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdf64001690 con 0x7fdf7409c870 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf7c206640 5 monclient: authenticate success, global_id 4147 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf7c206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient: dump: 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf7c206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf64001870 con 0x7fdf7409c870 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 1 -- 172.21.15.196:0/3640697731 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdf64001c80 con 0x7fdf7409c870 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf7c206640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf7c206640 4 set_mon_vals no callback set 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.637+0000 7fdf78fa0640 10 monclient: dump: 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.638+0000 7fdf7c206640 10 monclient: shutdown 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.638+0000 7fdf7c206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.638+0000 7fdf7c206640 1 -- 172.21.15.196:0/3640697731 >> v1:172.21.15.196:6789/0 conn(0x7fdf7409c870 legacy=0x7fdf7413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.638+0000 7fdf7c206640 1 -- 172.21.15.196:0/3640697731 shutdown_connections 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.638+0000 7fdf7c206640 1 -- 172.21.15.196:0/3640697731 >> 172.21.15.196:0/3640697731 conn(0x7fdf74066b00 msgr2=0x7fdf74066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.638+0000 7fdf7c206640 1 -- 172.21.15.196:0/3640697731 shutdown_connections 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.638+0000 7fdf7c206640 1 -- 172.21.15.196:0/3640697731 wait complete. 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.638+0000 7fdf7c206640 10 monclient: build_initial_monmap 2024-03-27T03:05:34.646 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.638+0000 7fdf7c206640 10 monclient: monmap: 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 1 librados: starting msgr at 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 1 librados: starting objecter 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 1 Processor -- start 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 1 -- start start 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 1 librados: setting wanted keys 2024-03-27T03:05:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 1 librados: calling monclient init 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 10 monclient: init 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 10 monclient(hunting): picked mon.noname-c con 0x7fdf7409ac40 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 10 monclient(hunting): picked mon.noname-b con 0x7fdf740af290 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 10 monclient(hunting): picked mon.noname-a con 0x7fdf74139400 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740b1680 con 0x7fdf74139400 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740b0860 con 0x7fdf740af290 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf741399c0 con 0x7fdf7409ac40 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf7c206640 10 monclient(hunting): authenticate will time out at 815.958471s 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf797a1640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdf740af290 0x7fdf74138cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:34798/0 (socket says 172.21.15.196:34798) 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf797a1640 1 -- 172.21.15.196:0/4251411061 learned_addr learned my addr 172.21.15.196:0/4251411061 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf6a7fc640 10 client.?.objecter ms_handle_connect 0x7fdf740af290 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.648 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf6a7fc640 10 client.?.objecter ms_handle_connect 0x7fdf74139400 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf6a7fc640 10 client.?.objecter ms_handle_connect 0x7fdf7409ac40 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.639+0000 7fdf6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172644155 0 0) 0x7fdf740b0860 con 0x7fdf740af290 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): my global_id is 4150 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf58004eb0 con 0x7fdf740af290 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651755551 0 0) 0x7fdf740b1680 con 0x7fdf74139400 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): my global_id is 4149 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf740b0860 con 0x7fdf74139400 2024-03-27T03:05:34.649 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804751093 0 0) 0x7fdf741399c0 con 0x7fdf7409ac40 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): my global_id is 4163 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf740b1680 con 0x7fdf7409ac40 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1917185738 0 0) 0x7fdf58004eb0 con 0x7fdf740af290 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf741399c0 con 0x7fdf740af290 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdf64001bc0 con 0x7fdf740af290 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): dump: 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.650 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2667055323 0 0) 0x7fdf740b0860 con 0x7fdf74139400 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf58004eb0 con 0x7fdf74139400 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdf6c001a80 con 0x7fdf74139400 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): dump: 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2590833481 0 0) 0x7fdf740b1680 con 0x7fdf7409ac40 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf740b0860 con 0x7fdf7409ac40 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdf70001870 con 0x7fdf7409ac40 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient(hunting): dump: 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.652 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 56986085 0 0) 0x7fdf741399c0 con 0x7fdf740af290 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 >> v1:172.21.15.153:6790/0 conn(0x7fdf7409ac40 legacy=0x7fdf740aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 >> v1:172.21.15.153:6789/0 conn(0x7fdf74139400 legacy=0x7fdf741da840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 monclient: found mon.b 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf741daf50 con 0x7fdf740af290 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient: _finish_auth 0 2024-03-27T03:05:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf7c206640 5 monclient: authenticate success, global_id 4150 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf7c206640 10 monclient: _renew_subs 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf7c206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf7c206640 1 -- 172.21.15.196:0/4251411061 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf741db1a0 con 0x7fdf740af290 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf7c206640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf7c206640 10 monclient: _renew_subs 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf7c206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf7c206640 1 -- 172.21.15.196:0/4251411061 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf741e3910 con 0x7fdf740af290 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf7c206640 1 librados: init done 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf64004bf0 con 0x7fdf740af290 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdf64004f00 con 0x7fdf740af290 2024-03-27T03:05:34.654 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 monclient: dump: 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6bfff640 4 set_mon_vals no callback set 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdf64005470 con 0x7fdf740af290 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 474496521 0 0) 0x7fdf64006bc0 con 0x7fdf740af290 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 10 client.4150.objecter ms_dispatch 0x7fdf74000b90 osd_map(9..9 src has 1..9) v4 2024-03-27T03:05:34.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 3 client.4150.objecter handle_osd_map got epochs [9,9] > 0 2024-03-27T03:05:34.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 3 client.4150.objecter handle_osd_map decoding full epoch 9 2024-03-27T03:05:34.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.640+0000 7fdf6a7fc640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-27T03:05:34.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.641+0000 7fdf7c206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.641+0000 7fdf7c206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.641+0000 7fdf7c206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.641+0000 7fdf7c206640 1 -- 172.21.15.196:0/4251411061 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5c005080 con 0x7fdf740af290 2024-03-27T03:05:34.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.643+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdf64006610 con 0x7fdf740af290 2024-03-27T03:05:34.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.643+0000 7fdf6a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.643+0000 7fdf6a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:34.666 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:34.673+0000 7f76d4a21040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-27T03:05:34.683 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:34.690+0000 7f9854a42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-27T03:05:34.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.715+0000 7fdf7c206640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "47e2cc28-454d-43f2-b42c-d3b267a87d2c", "id": 7}] 2024-03-27T03:05:34.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.715+0000 7fdf7c206640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "47e2cc28-454d-43f2-b42c-d3b267a87d2c", "id": 7}] 2024-03-27T03:05:34.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.715+0000 7fdf7c206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:34.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.715+0000 7fdf7c206640 1 -- 172.21.15.196:0/4251411061 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "47e2cc28-454d-43f2-b42c-d3b267a87d2c", "id": 7} v 0) v1 -- 0x7fdf5c0058b0 con 0x7fdf740af290 2024-03-27T03:05:34.712 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.719+0000 7fdf6a7fc640 1 -- 172.21.15.196:0/4251411061 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "47e2cc28-454d-43f2-b42c-d3b267a87d2c", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2446744189 0 1371665406) 0x7fdf64006880 con 0x7fdf740af290 2024-03-27T03:05:34.712 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.719+0000 7fdf6a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "47e2cc28-454d-43f2-b42c-d3b267a87d2c", "id": 7}] 2024-03-27T03:05:34.712 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.719+0000 7fdf6a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:34.712 INFO:teuthology.orchestra.run.smithi196.stdout:7 2024-03-27T03:05:34.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.720+0000 7fdf7c206640 10 librados: watch_flush enter 2024-03-27T03:05:34.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.720+0000 7fdf7c206640 10 librados: watch_flush exit 2024-03-27T03:05:34.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.720+0000 7fdf7c206640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-27T03:05:34.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.720+0000 7fdf7c206640 10 client.4150.objecter successfully canceled tick 2024-03-27T03:05:34.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.720+0000 7fdf7c206640 10 monclient: shutdown 2024-03-27T03:05:34.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.720+0000 7fdf7c206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:34.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.720+0000 7fdf7c206640 1 -- 172.21.15.196:0/4251411061 >> v1:172.21.15.196:6789/0 conn(0x7fdf740af290 legacy=0x7fdf74138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.720+0000 7fdf7c206640 1 -- 172.21.15.196:0/4251411061 shutdown_connections 2024-03-27T03:05:34.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.720+0000 7fdf7c206640 1 -- 172.21.15.196:0/4251411061 >> 172.21.15.196:0/4251411061 conn(0x7fdf74066b00 msgr2=0x7fdf7413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:34.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.720+0000 7fdf7c206640 1 -- 172.21.15.196:0/4251411061 shutdown_connections 2024-03-27T03:05:34.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.720+0000 7fdf7c206640 1 -- 172.21.15.196:0/4251411061 wait complete. 2024-03-27T03:05:34.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-27T03:05:34.720+0000 7fdf7c206640 1 librados: shutdown 2024-03-27T03:05:34.714 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:34.719+0000 7f76d4a21040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-27T03:05:34.723 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-27T03:05:34.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-27T03:05:34.725 INFO:tasks.ceph.osd.0:Started 2024-03-27T03:05:34.726 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-27T03:05:34.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-27T03:05:34.727 INFO:tasks.ceph.osd.1:Started 2024-03-27T03:05:34.728 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-27T03:05:34.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-27T03:05:34.730 INFO:tasks.ceph.osd.2:Started 2024-03-27T03:05:34.730 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-27T03:05:34.730 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-27T03:05:34.732 INFO:tasks.ceph.osd.3:Started 2024-03-27T03:05:34.732 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-27T03:05:34.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-27T03:05:34.766 INFO:tasks.ceph.osd.4:Started 2024-03-27T03:05:34.766 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-27T03:05:34.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-27T03:05:34.769 INFO:tasks.ceph.osd.5:Started 2024-03-27T03:05:34.769 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-27T03:05:34.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-27T03:05:34.771 INFO:tasks.ceph.osd.6:Started 2024-03-27T03:05:34.771 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-27T03:05:34.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-27T03:05:34.773 INFO:tasks.ceph.osd.7:Started 2024-03-27T03:05:34.773 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T03:05:34.842 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:34.848+0000 7f9854a42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-27T03:05:34.863 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:34.870+0000 7f76d4a21040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.897+0000 7f2c0324a640 10 monclient: get_monmap_and_config 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.897+0000 7f2c0324a640 10 monclient: build_initial_monmap 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.897+0000 7f2c0324a640 10 monclient: monmap: 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-27T03:05:34.898493+0000 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-27T03:05:34.898493+0000 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:34.891 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:34.898 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:34.901+0000 7f9854a42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-27T03:05:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.898+0000 7f2c0324a640 1 Processor -- start 2024-03-27T03:05:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.898+0000 7f2c0324a640 1 -- start start 2024-03-27T03:05:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.898+0000 7f2c0324a640 10 monclient: init 2024-03-27T03:05:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.898+0000 7f2c0324a640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.898+0000 7f2c0324a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.898+0000 7f2c0324a640 10 monclient(hunting): picked mon.noname-b con 0x7f2bfc0b8240 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.898+0000 7f2c0324a640 10 monclient(hunting): picked mon.noname-c con 0x7f2bfc0b1f60 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.898+0000 7f2c0324a640 10 monclient(hunting): picked mon.noname-a con 0x7f2bfc09c8c0 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.898+0000 7f2c0324a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc09bd80 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2c0324a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc09b960 con 0x7f2bfc0b1f60 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2c0324a640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc0b2ec0 con 0x7f2bfc0b8240 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2c0324a640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2c0324a640 10 monclient(hunting): authenticate will time out at 1295.204720s 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2c017e7640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f2bfc09c8c0 0x7f2bfc14d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:52272/0 (socket says 172.21.15.153:52272) 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2c00fe6640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f2bfc0b8240 0x7f2bfc13f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:46360/0 (socket says 172.21.15.153:46360) 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2c017e7640 1 -- 172.21.15.153:0/3008172887 learned_addr learned my addr 172.21.15.153:0/3008172887 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:05:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793810073 0 0) 0x7f2bfc0b2ec0 con 0x7f2bfc0b8240 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient(hunting): my global_id is 4190 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bec003520 con 0x7f2bfc0b8240 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517285042 0 0) 0x7f2bfc09bd80 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient(hunting): my global_id is 4176 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bfc0b2ec0 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14173541 0 0) 0x7f2bfc09b960 con 0x7f2bfc0b1f60 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient(hunting): my global_id is 4177 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bfc09bd80 con 0x7f2bfc0b1f60 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3338335569 0 0) 0x7f2bec003520 con 0x7f2bfc0b8240 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 >> v1:172.21.15.196:6789/0 conn(0x7f2bfc0b1f60 legacy=0x7f2bfc09c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 >> v1:172.21.15.153:6789/0 conn(0x7f2bfc09c8c0 legacy=0x7f2bfc14d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient: _finish_hunting 0 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 monclient: found mon.noname-b 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc13fb60 con 0x7f2bfc0b8240 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient: _finish_auth 0 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient: _check_auth_tickets 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2bf0001690 con 0x7f2bfc0b8240 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2c0324a640 5 monclient: authenticate success, global_id 4190 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2c0324a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient: dump: 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:34.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2c0324a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bf00017f0 con 0x7f2bfc0b8240 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 1 -- 172.21.15.153:0/3008172887 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2bf0001c40 con 0x7f2bfc0b8240 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2bfb7fe640 10 monclient: dump: 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2c0324a640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.899+0000 7f2c0324a640 4 set_mon_vals no callback set 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.900+0000 7f2c0324a640 10 monclient: shutdown 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.900+0000 7f2c0324a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.900+0000 7f2c0324a640 1 -- 172.21.15.153:0/3008172887 >> v1:172.21.15.153:6790/0 conn(0x7f2bfc0b8240 legacy=0x7f2bfc13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.900+0000 7f2c0324a640 1 -- 172.21.15.153:0/3008172887 shutdown_connections 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.900+0000 7f2c0324a640 1 -- 172.21.15.153:0/3008172887 >> 172.21.15.153:0/3008172887 conn(0x7f2bfc066b00 msgr2=0x7f2bfc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.900+0000 7f2c0324a640 1 -- 172.21.15.153:0/3008172887 shutdown_connections 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.900+0000 7f2c0324a640 1 -- 172.21.15.153:0/3008172887 wait complete. 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.901+0000 7f2c0324a640 10 monclient: build_initial_monmap 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.901+0000 7f2c0324a640 10 monclient: monmap: 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-27T03:05:34.906 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.901+0000 7f2c0324a640 1 librados: starting msgr at 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.901+0000 7f2c0324a640 1 librados: starting objecter 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.901+0000 7f2c0324a640 1 Processor -- start 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.901+0000 7f2c0324a640 1 -- start start 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.901+0000 7f2c0324a640 1 librados: setting wanted keys 2024-03-27T03:05:34.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.901+0000 7f2c0324a640 1 librados: calling monclient init 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.901+0000 7f2c0324a640 10 monclient: init 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2c0324a640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2c0324a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2c0324a640 10 monclient(hunting): picked mon.noname-c con 0x7f2bfc09c8c0 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2c0324a640 10 monclient(hunting): picked mon.noname-a con 0x7f2bfc0b8240 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2c0324a640 10 monclient(hunting): picked mon.noname-b con 0x7f2bfc06d3f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2c0324a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc1db970 con 0x7f2bfc0b8240 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2c0324a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc06e810 con 0x7f2bfc06d3f0 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2c0324a640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc06e9a0 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2c0324a640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2c0324a640 10 monclient(hunting): authenticate will time out at 1295.207855s 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2c00fe6640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f2bfc09c8c0 0x7f2bfc06e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:46374/0 (socket says 172.21.15.153:46374) 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2c00fe6640 1 -- 172.21.15.153:0/356307162 learned_addr learned my addr 172.21.15.153:0/356307162 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 client.?.objecter ms_handle_connect 0x7f2bfc09c8c0 2024-03-27T03:05:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 client.?.objecter ms_handle_connect 0x7f2bfc0b8240 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310407379 0 0) 0x7f2bfc1db970 con 0x7f2bfc0b8240 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): my global_id is 4179 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be0004eb0 con 0x7f2bfc0b8240 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 client.?.objecter ms_handle_connect 0x7f2bfc06d3f0 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924311442 0 0) 0x7f2bfc06e9a0 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): my global_id is 4193 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bfc1db970 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30523527 0 0) 0x7f2bfc06e810 con 0x7f2bfc06d3f0 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): _negotiate 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): my global_id is 4180 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bfc06e9a0 con 0x7f2bfc06d3f0 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2375844706 0 0) 0x7f2be0004eb0 con 0x7f2bfc0b8240 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bfc06e810 con 0x7f2bfc0b8240 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1229664214 0 0) 0x7f2bfc1db970 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2be0004eb0 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2be80018f0 con 0x7f2bfc0b8240 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): dump: 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2bf0001bc0 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:34.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.902+0000 7f2bf97fa640 10 monclient(hunting): dump: 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2324774617 0 0) 0x7f2be0004eb0 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 >> v1:172.21.15.196:6789/0 conn(0x7f2bfc06d3f0 legacy=0x7f2bfc1e60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 >> v1:172.21.15.153:6789/0 conn(0x7f2bfc0b8240 legacy=0x7f2bfc1dbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 10 monclient: _finish_hunting 0 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 1 monclient: found mon.c 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc1dc580 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 10 monclient: _finish_auth 0 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 10 monclient: _check_auth_tickets 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2c0324a640 5 monclient: authenticate success, global_id 4193 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2c0324a640 10 monclient: _renew_subs 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2c0324a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2c0324a640 1 -- 172.21.15.153:0/356307162 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bfc070df0 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2c0324a640 10 client.4193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2c0324a640 10 monclient: _renew_subs 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2c0324a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2c0324a640 1 -- 172.21.15.153:0/356307162 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bfc1e79c0 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bf0001d20 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2c0324a640 1 librados: init done 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2bf0004d90 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 10 monclient: dump: 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bfaffd640 4 set_mon_vals no callback set 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2bf0005300 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f2bf0005c80 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 10 client.4193.objecter ms_dispatch 0x7f2bfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 3 client.4193.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 3 client.4193.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.903+0000 7f2bf97fa640 20 client.4193.objecter dump_active .. 0 homeless 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.904+0000 7f2c0324a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.904+0000 7f2c0324a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.904+0000 7f2c0324a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.904+0000 7f2c0324a640 1 -- 172.21.15.153:0/356307162 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bfc09b170 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.906+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2bf0005e40 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.906+0000 7f2bf97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.906+0000 7f2bf97fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:34.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.967+0000 7f2c0324a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:34.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.967+0000 7f2c0324a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:34.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.967+0000 7f2c0324a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:05:34.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.967+0000 7f2c0324a640 1 -- 172.21.15.153:0/356307162 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bfc06f7c0 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.967+0000 7f2bf97fa640 1 -- 172.21.15.153:0/356307162 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f2bf00055b0 con 0x7f2bfc09c8c0 2024-03-27T03:05:34.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.967+0000 7f2bf97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:34.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.967+0000 7f2bf97fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:34.960 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-27T03:05:34.961 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:05:34.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.968+0000 7f2c0324a640 10 librados: watch_flush enter 2024-03-27T03:05:34.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.968+0000 7f2c0324a640 10 librados: watch_flush exit 2024-03-27T03:05:34.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.968+0000 7f2c0324a640 20 client.4193.objecter shutdown clearing up homeless session... 2024-03-27T03:05:34.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.968+0000 7f2c0324a640 10 client.4193.objecter successfully canceled tick 2024-03-27T03:05:34.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.968+0000 7f2c0324a640 10 monclient: shutdown 2024-03-27T03:05:34.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.968+0000 7f2c0324a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:34.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.968+0000 7f2c0324a640 1 -- 172.21.15.153:0/356307162 >> v1:172.21.15.153:6790/0 conn(0x7f2bfc09c8c0 legacy=0x7f2bfc06e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:34.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.968+0000 7f2c0324a640 1 -- 172.21.15.153:0/356307162 shutdown_connections 2024-03-27T03:05:34.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.968+0000 7f2c0324a640 1 -- 172.21.15.153:0/356307162 >> 172.21.15.153:0/356307162 conn(0x7f2bfc066b00 msgr2=0x7f2bfc069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:34.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.968+0000 7f2c0324a640 1 -- 172.21.15.153:0/356307162 shutdown_connections 2024-03-27T03:05:34.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.968+0000 7f2c0324a640 1 -- 172.21.15.153:0/356307162 wait complete. 2024-03-27T03:05:34.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-27T03:05:34.968+0000 7f2c0324a640 1 librados: shutdown 2024-03-27T03:05:34.963 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:34.970+0000 7f76d4a21040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-27T03:05:34.972 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-27T03:05:34.972 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-27T03:05:35.004 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:35.011+0000 7f9854a42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-27T03:05:35.098 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:35.104+0000 7f9854a42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-27T03:05:35.121 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:35.129+0000 7f76d4a21040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-27T03:05:35.164 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:35.171+0000 7f76d4a21040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-27T03:05:35.250 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:35.257+0000 7f9854a42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-27T03:05:35.292 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:35.299+0000 7f9854a42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-27T03:05:35.318 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:35.325+0000 7f76d4a21040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-27T03:05:35.322 INFO:tasks.ceph.osd.0.smithi153.stderr:2024-03-27T03:05:35.329+0000 7f07b7423640 -1 Falling back to public interface 2024-03-27T03:05:35.331 INFO:tasks.ceph.osd.1.smithi153.stderr:2024-03-27T03:05:35.338+0000 7f9413413640 -1 Falling back to public interface 2024-03-27T03:05:35.339 INFO:tasks.ceph.osd.3.smithi153.stderr:2024-03-27T03:05:35.347+0000 7f02cd0d9640 -1 Falling back to public interface 2024-03-27T03:05:35.343 INFO:tasks.ceph.osd.2.smithi153.stderr:2024-03-27T03:05:35.350+0000 7f85f4021640 -1 Falling back to public interface 2024-03-27T03:05:35.364 INFO:tasks.ceph.osd.4.smithi196.stderr:2024-03-27T03:05:35.371+0000 7f08030d3640 -1 Falling back to public interface 2024-03-27T03:05:35.369 INFO:tasks.ceph.osd.5.smithi196.stderr:2024-03-27T03:05:35.376+0000 7febace19640 -1 Falling back to public interface 2024-03-27T03:05:35.371 INFO:tasks.ceph.osd.6.smithi196.stderr:2024-03-27T03:05:35.378+0000 7f28c2a0c640 -1 Falling back to public interface 2024-03-27T03:05:35.386 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-03-27T03:05:35.394+0000 7fd365c0f640 -1 Falling back to public interface 2024-03-27T03:05:35.449 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:35.456+0000 7f9854a42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-27T03:05:35.778 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:05:35.868 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.875+0000 7fc0055f4640 10 monclient: get_monmap_and_config 2024-03-27T03:05:35.868 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.875+0000 7fc0055f4640 10 monclient: build_initial_monmap 2024-03-27T03:05:35.868 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.875+0000 7fc0055f4640 10 monclient: monmap: 2024-03-27T03:05:35.869 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:05:35.869 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:35.869 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:05:35.876622+0000 2024-03-27T03:05:35.869 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:05:35.876622+0000 2024-03-27T03:05:35.869 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:35.869 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:35.869 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:35.869 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:35.869 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:35.869 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:35.877 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.876+0000 7fc0055f4640 1 Processor -- start 2024-03-27T03:05:35.877 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.876+0000 7fc0055f4640 1 -- start start 2024-03-27T03:05:35.877 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.876+0000 7fc0055f4640 10 monclient: init 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 10 monclient(hunting): picked mon.noname-b con 0x7fc0001457f0 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 10 monclient(hunting): picked mon.noname-c con 0x7fc00013f650 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 10 monclient(hunting): picked mon.noname-a con 0x7fc0000a9d70 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0000aaef0 con 0x7fc0000a9d70 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc000145bd0 con 0x7fc00013f650 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc000145eb0 con 0x7fc0001457f0 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 10 monclient(hunting): authenticate will time out at 1296.182707s 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffeffd640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fc0001457f0 0x7fc00013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:46390/0 (socket says 172.21.15.153:46390) 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffeffd640 1 -- 172.21.15.153:0/163528091 learned_addr learned my addr 172.21.15.153:0/163528091 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:05:35.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771365779 0 0) 0x7fc000145eb0 con 0x7fc0001457f0 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient(hunting): my global_id is 4196 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfec003520 con 0x7fc0001457f0 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544396391 0 0) 0x7fc0000aaef0 con 0x7fc0000a9d70 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient(hunting): my global_id is 4182 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc000145eb0 con 0x7fc0000a9d70 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762973790 0 0) 0x7fc000145bd0 con 0x7fc00013f650 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:35.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient(hunting): my global_id is 4183 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0000aaef0 con 0x7fc00013f650 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4292314076 0 0) 0x7fc000145eb0 con 0x7fc0000a9d70 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 >> v1:172.21.15.153:6790/0 conn(0x7fc0001457f0 legacy=0x7fc00013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 >> v1:172.21.15.196:6789/0 conn(0x7fc00013f650 legacy=0x7fc0000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 monclient: found mon.noname-a 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc000146340 con 0x7fc0000a9d70 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient: _finish_auth 0 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fbff80019f0 con 0x7fc0000a9d70 2024-03-27T03:05:35.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient: dump: 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:35.881 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 5 monclient: authenticate success, global_id 4182 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbff8001bd0 con 0x7fc0000a9d70 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 1 -- 172.21.15.153:0/163528091 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fbff8002020 con 0x7fc0000a9d70 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fc0055f4640 4 set_mon_vals no callback set 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.877+0000 7fbffe7fc640 10 monclient: dump: 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:35.882 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.878+0000 7fc0055f4640 10 monclient: shutdown 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.878+0000 7fc0055f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.878+0000 7fc0055f4640 1 -- 172.21.15.153:0/163528091 >> v1:172.21.15.153:6789/0 conn(0x7fc0000a9d70 legacy=0x7fc00014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.878+0000 7fc0055f4640 1 -- 172.21.15.153:0/163528091 shutdown_connections 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.878+0000 7fc0055f4640 1 -- 172.21.15.153:0/163528091 >> 172.21.15.153:0/163528091 conn(0x7fc000066a00 msgr2=0x7fc000066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.878+0000 7fc0055f4640 1 -- 172.21.15.153:0/163528091 shutdown_connections 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.878+0000 7fc0055f4640 1 -- 172.21.15.153:0/163528091 wait complete. 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.879+0000 7fc0055f4640 10 monclient: build_initial_monmap 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.879+0000 7fc0055f4640 10 monclient: monmap: 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:35.883 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.879+0000 7fc0055f4640 1 librados: starting msgr at 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.879+0000 7fc0055f4640 1 librados: starting objecter 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.879+0000 7fc0055f4640 1 Processor -- start 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.879+0000 7fc0055f4640 1 -- start start 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.879+0000 7fc0055f4640 1 librados: setting wanted keys 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.879+0000 7fc0055f4640 1 librados: calling monclient init 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.879+0000 7fc0055f4640 10 monclient: init 2024-03-27T03:05:35.884 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fc0055f4640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fc0055f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fc0055f4640 10 monclient(hunting): picked mon.noname-c con 0x7fc0000a9d70 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fc0055f4640 10 monclient(hunting): picked mon.noname-a con 0x7fc00013f650 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fc0055f4640 10 monclient(hunting): picked mon.noname-b con 0x7fc0001457f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fc0055f4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc000138a90 con 0x7fc00013f650 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fc0055f4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc000135ba0 con 0x7fc0001457f0 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fc0055f4640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00013a5b0 con 0x7fc0000a9d70 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fc0055f4640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fc0055f4640 10 monclient(hunting): authenticate will time out at 1296.185795s 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff7fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc00013f650 0x7fc000134d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:52308/0 (socket says 172.21.15.153:52308) 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff7fff640 1 -- 172.21.15.153:0/2431785555 learned_addr learned my addr 172.21.15.153:0/2431785555 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 client.?.objecter ms_handle_connect 0x7fc0000a9d70 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 client.?.objecter ms_handle_connect 0x7fc00013f650 2024-03-27T03:05:35.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 client.?.objecter ms_handle_connect 0x7fc0001457f0 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525254903 0 0) 0x7fc00013a5b0 con 0x7fc0000a9d70 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): _negotiate 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): my global_id is 4199 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfe0004eb0 con 0x7fc0000a9d70 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864457874 0 0) 0x7fc000138a90 con 0x7fc00013f650 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): _negotiate 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): my global_id is 4185 2024-03-27T03:05:35.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc00013a5b0 con 0x7fc00013f650 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664950190 0 0) 0x7fc000135ba0 con 0x7fc0001457f0 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): _negotiate 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): my global_id is 4186 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc000138a90 con 0x7fc0001457f0 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3918046310 0 0) 0x7fc000138a90 con 0x7fc0001457f0 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc000135ba0 con 0x7fc0001457f0 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fbff8001ee0 con 0x7fc0001457f0 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): dump: 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:35.887 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3970148885 0 0) 0x7fbfe0004eb0 con 0x7fc0000a9d70 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc000138a90 con 0x7fc0000a9d70 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2615747693 0 0) 0x7fc00013a5b0 con 0x7fc00013f650 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfe0004eb0 con 0x7fc00013f650 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fbff00016f0 con 0x7fc0000a9d70 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:35.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): dump: 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fbfe80018f0 con 0x7fc00013f650 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:05:35.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.880+0000 7fbff77fe640 10 monclient(hunting): dump: 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2506501858 0 0) 0x7fc000135ba0 con 0x7fc0001457f0 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 >> v1:172.21.15.153:6790/0 conn(0x7fc0000a9d70 legacy=0x7fc000139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 >> v1:172.21.15.153:6789/0 conn(0x7fc00013f650 legacy=0x7fc000134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 10 monclient: _finish_hunting 0 2024-03-27T03:05:35.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 1 monclient: found mon.b 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00013a890 con 0x7fc0001457f0 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 10 monclient: _finish_auth 0 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 10 monclient: _check_auth_tickets 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fc0055f4640 5 monclient: authenticate success, global_id 4186 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fc0055f4640 10 monclient: _renew_subs 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fc0055f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fc0055f4640 1 -- 172.21.15.153:0/2431785555 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc00013ab70 con 0x7fc0001457f0 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fc0055f4640 10 client.4186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fc0055f4640 10 monclient: _renew_subs 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fc0055f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:35.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fc0055f4640 1 -- 172.21.15.153:0/2431785555 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc000142d10 con 0x7fc0001457f0 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fc0055f4640 1 librados: init done 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbff8001360 con 0x7fc0001457f0 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fbff8003e80 con 0x7fc0001457f0 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbffdffb640 4 set_mon_vals no callback set 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 10 monclient: dump: 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:35.892 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbff8004370 con 0x7fc0001457f0 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fbff8005ab0 con 0x7fc0001457f0 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 10 client.4186.objecter ms_dispatch 0x7fc000000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 3 client.4186.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 3 client.4186.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.881+0000 7fbff77fe640 20 client.4186.objecter dump_active .. 0 homeless 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.882+0000 7fc0055f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.882+0000 7fc0055f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.882+0000 7fc0055f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.882+0000 7fc0055f4640 1 -- 172.21.15.153:0/2431785555 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe4005080 con 0x7fc0001457f0 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.884+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbff8005490 con 0x7fc0001457f0 2024-03-27T03:05:35.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.884+0000 7fbff77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:35.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.884+0000 7fbff77fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.946+0000 7fc0055f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.946+0000 7fc0055f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.946+0000 7fc0055f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.946+0000 7fc0055f4640 1 -- 172.21.15.153:0/2431785555 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfe4005330 con 0x7fc0001457f0 2024-03-27T03:05:35.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.946+0000 7fbff77fe640 1 -- 172.21.15.153:0/2431785555 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fbff8005760 con 0x7fc0001457f0 2024-03-27T03:05:35.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.946+0000 7fbff77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:35.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.946+0000 7fbff77fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:35.940 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:05:35.940 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:05:35.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.947+0000 7fc0055f4640 10 librados: watch_flush enter 2024-03-27T03:05:35.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.947+0000 7fc0055f4640 10 librados: watch_flush exit 2024-03-27T03:05:35.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.947+0000 7fc0055f4640 20 client.4186.objecter shutdown clearing up homeless session... 2024-03-27T03:05:35.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.947+0000 7fc0055f4640 10 client.4186.objecter successfully canceled tick 2024-03-27T03:05:35.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.947+0000 7fc0055f4640 10 monclient: shutdown 2024-03-27T03:05:35.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.947+0000 7fc0055f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:35.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.947+0000 7fc0055f4640 1 -- 172.21.15.153:0/2431785555 >> v1:172.21.15.196:6789/0 conn(0x7fc0001457f0 legacy=0x7fc000135490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:35.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.947+0000 7fc0055f4640 1 -- 172.21.15.153:0/2431785555 shutdown_connections 2024-03-27T03:05:35.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.947+0000 7fc0055f4640 1 -- 172.21.15.153:0/2431785555 >> 172.21.15.153:0/2431785555 conn(0x7fc000066a00 msgr2=0x7fc0000b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:35.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.948+0000 7fc0055f4640 1 -- 172.21.15.153:0/2431785555 shutdown_connections 2024-03-27T03:05:35.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.948+0000 7fc0055f4640 1 -- 172.21.15.153:0/2431785555 wait complete. 2024-03-27T03:05:35.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:35.948+0000 7fc0055f4640 1 librados: shutdown 2024-03-27T03:05:35.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:05:36.145 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:36.152+0000 7f76d4a21040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-27T03:05:36.416 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:36.423+0000 7f76d4a21040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-27T03:05:36.469 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:36.476+0000 7f76d4a21040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-27T03:05:36.569 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:36.576+0000 7f9854a42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-27T03:05:36.583 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:36.590+0000 7f76d4a21040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-27T03:05:36.847 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:36.854+0000 7f9854a42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-27T03:05:36.897 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:36.904+0000 7f9854a42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-27T03:05:37.012 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-27T03:05:37.019+0000 7f9854a42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-27T03:05:37.610 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:37.617+0000 7f76753dc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-27T03:05:37.611 INFO:tasks.ceph.mgr.y.smithi153.stderr:2024-03-27T03:05:37.618+0000 7f76753dc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-27T03:05:38.799 INFO:tasks.ceph.osd.1.smithi153.stderr:2024-03-27T03:05:38.806+0000 7f9413413640 -1 osd.1 0 log_to_monitors true 2024-03-27T03:05:38.823 INFO:tasks.ceph.osd.4.smithi196.stderr:2024-03-27T03:05:38.830+0000 7f08030d3640 -1 osd.4 0 log_to_monitors true 2024-03-27T03:05:38.862 INFO:tasks.ceph.osd.3.smithi153.stderr:2024-03-27T03:05:38.869+0000 7f02cd0d9640 -1 osd.3 0 log_to_monitors true 2024-03-27T03:05:38.867 INFO:tasks.ceph.osd.5.smithi196.stderr:2024-03-27T03:05:38.874+0000 7febace19640 -1 osd.5 0 log_to_monitors true 2024-03-27T03:05:38.875 INFO:tasks.ceph.osd.6.smithi196.stderr:2024-03-27T03:05:38.883+0000 7f28c2a0c640 -1 osd.6 0 log_to_monitors true 2024-03-27T03:05:38.880 INFO:tasks.ceph.osd.2.smithi153.stderr:2024-03-27T03:05:38.887+0000 7f85f4021640 -1 osd.2 0 log_to_monitors true 2024-03-27T03:05:38.882 INFO:tasks.ceph.osd.0.smithi153.stderr:2024-03-27T03:05:38.889+0000 7f07b7423640 -1 osd.0 0 log_to_monitors true 2024-03-27T03:05:38.883 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-03-27T03:05:38.890+0000 7fd365c0f640 -1 osd.7 0 log_to_monitors true 2024-03-27T03:05:42.756 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:05:42.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.930+0000 7f648bf2b640 10 monclient: get_monmap_and_config 2024-03-27T03:05:42.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.930+0000 7f648bf2b640 10 monclient: build_initial_monmap 2024-03-27T03:05:42.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.930+0000 7f648bf2b640 10 monclient: monmap: 2024-03-27T03:05:42.924 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:05:42.924 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:42.924 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:05:42.931860+0000 2024-03-27T03:05:42.924 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:05:42.931860+0000 2024-03-27T03:05:42.924 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:42.924 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:42.924 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:42.924 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:42.925 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:42.925 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:42.925 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.932+0000 7f648bf2b640 1 Processor -- start 2024-03-27T03:05:42.925 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.932+0000 7f648bf2b640 1 -- start start 2024-03-27T03:05:42.925 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.932+0000 7f648bf2b640 10 monclient: init 2024-03-27T03:05:42.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f648bf2b640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:42.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f648bf2b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:05:42.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f648bf2b640 10 monclient(hunting): picked mon.noname-a con 0x7f64841422b0 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:42.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f648bf2b640 10 monclient(hunting): picked mon.noname-c con 0x7f6484142690 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:42.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f648bf2b640 10 monclient(hunting): picked mon.noname-b con 0x7f64840b17d0 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:42.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f648bf2b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6484142c80 con 0x7f64841422b0 2024-03-27T03:05:42.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f648bf2b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64840aae40 con 0x7f6484142690 2024-03-27T03:05:42.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f648bf2b640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f648413fbd0 con 0x7f64840b17d0 2024-03-27T03:05:42.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f648bf2b640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:42.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f648bf2b640 10 monclient(hunting): authenticate will time out at 1303.239185s 2024-03-27T03:05:42.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f6489cc7640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f64841422b0 0x7f648413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34580/0 (socket says 172.21.15.153:34580) 2024-03-27T03:05:42.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f6489cc7640 1 -- 172.21.15.153:0/538950607 learned_addr learned my addr 172.21.15.153:0/538950607 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:05:42.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.933+0000 7f648a4c8640 1 --1- 172.21.15.153:0/538950607 >> v1:172.21.15.153:6790/0 conn(0x7f64840b17d0 0x7f64840b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:58138/0 (socket says 172.21.15.153:58138) 2024-03-27T03:05:42.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.934+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358752465 0 0) 0x7f6484142c80 con 0x7f64841422b0 2024-03-27T03:05:42.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.934+0000 7f6488cc5640 10 monclient(hunting): _negotiate 2024-03-27T03:05:42.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.934+0000 7f6488cc5640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:42.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.934+0000 7f6488cc5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:42.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.934+0000 7f6488cc5640 10 monclient(hunting): my global_id is 4230 2024-03-27T03:05:42.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.934+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6470003520 con 0x7f64841422b0 2024-03-27T03:05:42.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.934+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592072055 0 0) 0x7f64840aae40 con 0x7f6484142690 2024-03-27T03:05:42.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.934+0000 7f6488cc5640 10 monclient(hunting): _negotiate 2024-03-27T03:05:42.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.934+0000 7f6488cc5640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:42.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.934+0000 7f6488cc5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:42.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.934+0000 7f6488cc5640 10 monclient(hunting): my global_id is 4225 2024-03-27T03:05:42.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.934+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6484142c80 con 0x7f6484142690 2024-03-27T03:05:42.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770765440 0 0) 0x7f648413fbd0 con 0x7f64840b17d0 2024-03-27T03:05:42.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 10 monclient(hunting): _negotiate 2024-03-27T03:05:42.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:42.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:42.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 10 monclient(hunting): my global_id is 4238 2024-03-27T03:05:42.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64840aae40 con 0x7f64840b17d0 2024-03-27T03:05:42.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 850573947 0 0) 0x7f6470003520 con 0x7f64841422b0 2024-03-27T03:05:42.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 >> v1:172.21.15.153:6790/0 conn(0x7f64840b17d0 legacy=0x7f64840b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:42.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 >> v1:172.21.15.196:6789/0 conn(0x7f6484142690 legacy=0x7f64840b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 10 monclient: _finish_hunting 0 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 1 monclient: found mon.noname-a 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6484151940 con 0x7f64841422b0 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 10 monclient: _finish_auth 0 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 10 monclient: _check_auth_tickets 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f648bf2b640 5 monclient: authenticate success, global_id 4230 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f647c001990 con 0x7f64841422b0 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f648bf2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f6488cc5640 10 monclient: dump: 2024-03-27T03:05:42.937 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:42.938 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:42.938 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.938 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.938 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:42.938 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:42.938 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:42.938 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:42.938 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:42.938 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:42.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.935+0000 7f648bf2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:42.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.936+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f647c001af0 con 0x7f64841422b0 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.936+0000 7f6488cc5640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.936+0000 7f6488cc5640 1 -- 172.21.15.153:0/538950607 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f647c001f40 con 0x7f64841422b0 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.936+0000 7f6488cc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.936+0000 7f6488cc5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.936+0000 7f6488cc5640 10 monclient: dump: 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:42.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.936+0000 7f648bf2b640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:42.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.936+0000 7f648bf2b640 4 set_mon_vals no callback set 2024-03-27T03:05:42.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.937+0000 7f648bf2b640 10 monclient: shutdown 2024-03-27T03:05:42.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.937+0000 7f648bf2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:42.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.937+0000 7f648bf2b640 1 -- 172.21.15.153:0/538950607 >> v1:172.21.15.153:6789/0 conn(0x7f64841422b0 legacy=0x7f648413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:42.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.937+0000 7f648bf2b640 1 -- 172.21.15.153:0/538950607 shutdown_connections 2024-03-27T03:05:42.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.937+0000 7f648bf2b640 1 -- 172.21.15.153:0/538950607 >> 172.21.15.153:0/538950607 conn(0x7f6484066a00 msgr2=0x7f6484066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:42.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.937+0000 7f648bf2b640 1 -- 172.21.15.153:0/538950607 shutdown_connections 2024-03-27T03:05:42.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.938+0000 7f648bf2b640 1 -- 172.21.15.153:0/538950607 wait complete. 2024-03-27T03:05:42.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.939+0000 7f648bf2b640 10 monclient: build_initial_monmap 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.939+0000 7f648bf2b640 10 monclient: monmap: 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.939+0000 7f648bf2b640 1 librados: starting msgr at 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.939+0000 7f648bf2b640 1 librados: starting objecter 2024-03-27T03:05:42.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.939+0000 7f648bf2b640 1 Processor -- start 2024-03-27T03:05:42.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.940+0000 7f648bf2b640 1 -- start start 2024-03-27T03:05:42.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.940+0000 7f648bf2b640 1 librados: setting wanted keys 2024-03-27T03:05:42.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.940+0000 7f648bf2b640 1 librados: calling monclient init 2024-03-27T03:05:42.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.940+0000 7f648bf2b640 10 monclient: init 2024-03-27T03:05:42.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f648bf2b640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:42.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f648bf2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:05:42.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f648bf2b640 10 monclient(hunting): picked mon.noname-a con 0x7f64840b17d0 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:42.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f648bf2b640 10 monclient(hunting): picked mon.noname-c con 0x7f64841422b0 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:42.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f6489cc7640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f64840b17d0 0x7f648412ff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34758/0 (socket says 172.21.15.153:34758) 2024-03-27T03:05:42.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f6489cc7640 1 -- 172.21.15.153:0/988346010 learned_addr learned my addr 172.21.15.153:0/988346010 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:05:42.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f648bf2b640 10 monclient(hunting): picked mon.noname-b con 0x7f6484142690 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64840ad290 con 0x7f64840b17d0 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64840ad570 con 0x7f6484142690 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f647a7fc640 10 client.?.objecter ms_handle_connect 0x7f64841422b0 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f647a7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f647a7fc640 10 client.?.objecter ms_handle_connect 0x7f6484142690 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.941+0000 7f647a7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64840ac080 con 0x7f64841422b0 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f648bf2b640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f648bf2b640 10 monclient(hunting): authenticate will time out at 1303.247739s 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 client.?.objecter ms_handle_connect 0x7f64840b17d0 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327540052 0 0) 0x7f64840ad570 con 0x7f6484142690 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:05:42.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 monclient(hunting): my global_id is 4228 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6468004eb0 con 0x7f6484142690 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341618122 0 0) 0x7f64840ac080 con 0x7f64841422b0 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 monclient(hunting): my global_id is 4241 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64840ad570 con 0x7f64841422b0 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253939691 0 0) 0x7f64840ad290 con 0x7f64840b17d0 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 10 monclient(hunting): my global_id is 4233 2024-03-27T03:05:42.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.942+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64840ac080 con 0x7f64840b17d0 2024-03-27T03:05:42.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 281887926 0 0) 0x7f64840ac080 con 0x7f64840b17d0 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64840ad290 con 0x7f64840b17d0 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f647c001e40 con 0x7f64840b17d0 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 10 monclient(hunting): dump: 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:42.947 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4234583573 0 0) 0x7f6468004eb0 con 0x7f6484142690 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64840ac080 con 0x7f6484142690 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 447504350 0 0) 0x7f64840ad570 con 0x7f64841422b0 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6468005e70 con 0x7f64841422b0 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6480001810 con 0x7f64841422b0 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 10 monclient(hunting): dump: 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.948 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6474001930 con 0x7f6484142690 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.943+0000 7f647a7fc640 10 monclient(hunting): dump: 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:42.949 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 622668997 0 0) 0x7f64840ac080 con 0x7f6484142690 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 >> v1:172.21.15.153:6790/0 conn(0x7f64841422b0 legacy=0x7f648412ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 >> v1:172.21.15.153:6789/0 conn(0x7f64840b17d0 legacy=0x7f648412ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 1 monclient: found mon.b 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64841327c0 con 0x7f6484142690 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 10 monclient: _finish_auth 0 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:42.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f648bf2b640 5 monclient: authenticate success, global_id 4228 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f648bf2b640 10 monclient: _renew_subs 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f648bf2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64840ac370 con 0x7f6484142690 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f648bf2b640 10 client.4228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f648bf2b640 10 monclient: _renew_subs 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f648bf2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6484130810 con 0x7f6484142690 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f648bf2b640 1 librados: init done 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64740047e0 con 0x7f6484142690 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6474001930 con 0x7f6484142690 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647a7fc640 10 monclient: dump: 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:42.951 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.944+0000 7f647bfff640 4 set_mon_vals no callback set 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.946+0000 7f648bf2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.946+0000 7f648bf2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.946+0000 7f648bf2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.946+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646c005080 con 0x7f6484142690 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.947+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f647401a7c0 con 0x7f6484142690 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.947+0000 7f6489cc7640 10 monclient: get_auth_request con 0x7f646806e560 auth_method 0 2024-03-27T03:05:42.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.948+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f6474054410 con 0x7f6484142690 2024-03-27T03:05:42.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.948+0000 7f647a7fc640 10 client.4228.objecter ms_dispatch 0x7f6484000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:05:42.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.948+0000 7f647a7fc640 3 client.4228.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:05:42.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.948+0000 7f647a7fc640 3 client.4228.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:05:42.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.948+0000 7f647a7fc640 20 client.4228.objecter dump_active .. 0 homeless 2024-03-27T03:05:42.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.948+0000 7f647a7fc640 10 client.4228.objecter ms_handle_connect 0x7f646806e560 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f648a4c8640 1 -- 172.21.15.153:0/988346010 >> v1:172.21.15.196:6789/0 conn(0x7f6484142690 legacy=0x7f64841e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f648a4c8640 1 -- 172.21.15.153:0/988346010 >> v1:172.21.15.196:6789/0 conn(0x7f6484142690 legacy=0x7f64841e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f648a4c8640 1 --1- 172.21.15.153:0/988346010 >> v1:172.21.15.196:6789/0 conn(0x7f6484142690 0x7f64841e2ee0 :-1 s=READ_MESSAGE_FRONT pgs=21347 cs=1 l=1).handle_message_data read data error 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f648a4c8640 1 --1- 172.21.15.153:0/988346010 >> v1:172.21.15.196:6789/0 conn(0x7f6484142690 0x7f64841e2ee0 :-1 s=READ_MESSAGE_FRONT pgs=21347 cs=1 l=1).fault on lossy channel, failing 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f647a7fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.196:6789/0 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f647a7fc640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 >> v1:172.21.15.196:6789/0 conn(0x7f6484142690 legacy=0x7f64841e2ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f647a7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f647a7fc640 10 monclient(hunting): picked mon.b con 0x7f6468073890 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f647a7fc640 10 monclient(hunting): picked mon.c con 0x7f64841422b0 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f647a7fc640 10 monclient(hunting): picked mon.a con 0x7f64840b17d0 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f64840ad570 con 0x7f64840b17d0 2024-03-27T03:05:42.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6468080200 con 0x7f6468073890 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f64680813c0 con 0x7f64841422b0 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.955+0000 7f647a7fc640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 client.4228.objecter ms_handle_connect 0x7f6468073890 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 client.4228.objecter resend_mon_ops 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 client.4228.objecter ms_handle_connect 0x7f64840b17d0 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 client.4228.objecter resend_mon_ops 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118338396 0 0) 0x7f6468080200 con 0x7f6468073890 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.196:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f646807f1d0 con 0x7f6468073890 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 client.4228.objecter ms_handle_connect 0x7f64841422b0 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 client.4228.objecter resend_mon_ops 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704025984 0 0) 0x7f64840ad570 con 0x7f64840b17d0 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:05:42.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.153:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f6468080200 con 0x7f64840b17d0 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2103324679 0 0) 0x7f646807f1d0 con 0x7f6468073890 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 >> v1:172.21.15.153:6790/0 conn(0x7f64841422b0 legacy=0x7f64680799b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 >> v1:172.21.15.153:6789/0 conn(0x7f64840b17d0 legacy=0x7f646807d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 1 monclient: found mon.b 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f64680826e0 con 0x7f6468073890 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6468080200 con 0x7f6468073890 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.956+0000 7f647a7fc640 10 monclient: _finish_auth 0 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.957+0000 7f647a7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:05:42.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.957+0000 7f647a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.957+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f64800035f0 con 0x7f6468073890 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.957+0000 7f647a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.957+0000 7f647a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.957+0000 7f647a7fc640 10 monclient: dump: 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:42.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.957+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6480003750 con 0x7f6468073890 2024-03-27T03:05:42.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.957+0000 7f647a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:42.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.957+0000 7f647b7fe640 4 set_mon_vals no callback set 2024-03-27T03:05:42.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.959+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f6480019730 con 0x7f6468073890 2024-03-27T03:05:42.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.963+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6480052640 con 0x7f6468073890 2024-03-27T03:05:42.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.963+0000 7f647a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:42.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:42.963+0000 7f647a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:43.339 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.346+0000 7f648bf2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:43.339 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.346+0000 7f648bf2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:43.339 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.346+0000 7f648bf2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:43.340 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.346+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f646c005330 con 0x7f6468073890 2024-03-27T03:05:43.340 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.347+0000 7f647a7fc640 1 -- 172.21.15.153:0/988346010 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f64800533e0 con 0x7f6468073890 2024-03-27T03:05:43.340 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.347+0000 7f647a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:43.340 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.347+0000 7f647a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:43.341 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:05:43.341 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:05:43.344 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.351+0000 7f648bf2b640 10 librados: watch_flush enter 2024-03-27T03:05:43.344 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.351+0000 7f648bf2b640 10 librados: watch_flush exit 2024-03-27T03:05:43.344 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.351+0000 7f648bf2b640 20 client.4228.objecter shutdown clearing up homeless session... 2024-03-27T03:05:43.344 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.351+0000 7f648bf2b640 10 client.4228.objecter successfully canceled tick 2024-03-27T03:05:43.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.351+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 >> v1:172.21.15.153:6816/2637937781 conn(0x7f646806e560 legacy=0x7f6468070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:43.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.351+0000 7f648bf2b640 10 monclient: shutdown 2024-03-27T03:05:43.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.351+0000 7f648bf2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:43.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.351+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 >> v1:172.21.15.196:6789/0 conn(0x7f6468073890 legacy=0x7f6468075ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:43.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.352+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 shutdown_connections 2024-03-27T03:05:43.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.352+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 >> 172.21.15.153:0/988346010 conn(0x7f6484066a00 msgr2=0x7f64840b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:43.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.352+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 shutdown_connections 2024-03-27T03:05:43.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.352+0000 7f648bf2b640 1 -- 172.21.15.153:0/988346010 wait complete. 2024-03-27T03:05:43.346 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:43.352+0000 7f648bf2b640 1 librados: shutdown 2024-03-27T03:05:43.358 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:05:50.166 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.308+0000 7fef81e7b640 10 monclient: get_monmap_and_config 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.308+0000 7fef81e7b640 10 monclient: build_initial_monmap 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.308+0000 7fef81e7b640 10 monclient: monmap: 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:05:50.308949+0000 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:05:50.308949+0000 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:50.301 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:50.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.309+0000 7fef81e7b640 1 Processor -- start 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.309+0000 7fef81e7b640 1 -- start start 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.309+0000 7fef81e7b640 10 monclient: init 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.309+0000 7fef81e7b640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.309+0000 7fef81e7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.309+0000 7fef81e7b640 10 monclient(hunting): picked mon.noname-c con 0x7fef7c0b4790 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.309+0000 7fef81e7b640 10 monclient(hunting): picked mon.noname-a con 0x7fef7c0aea90 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef81e7b640 10 monclient(hunting): picked mon.noname-b con 0x7fef7c0b39f0 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef81e7b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c0b0c60 con 0x7fef7c0aea90 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef81e7b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c0af020 con 0x7fef7c0b4790 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef7bfff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fef7c0b39f0 0x7fef7c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:39182/0 (socket says 172.21.15.153:39182) 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef7b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fef7c0b4790 0x7fef7c0b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:44552/0 (socket says 172.21.15.153:44552) 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef7bfff640 1 -- 172.21.15.153:0/307652229 learned_addr learned my addr 172.21.15.153:0/307652229 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef81e7b640 1 -- 172.21.15.153:0/307652229 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c13fbd0 con 0x7fef7c0b39f0 2024-03-27T03:05:50.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef81e7b640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:50.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef81e7b640 10 monclient(hunting): authenticate will time out at 1310.615903s 2024-03-27T03:05:50.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef7a7fc640 1 -- 172.21.15.153:0/307652229 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375043178 0 0) 0x7fef7c13fbd0 con 0x7fef7c0b39f0 2024-03-27T03:05:50.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef7a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:05:50.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:50.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:50.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef7a7fc640 10 monclient(hunting): my global_id is 4244 2024-03-27T03:05:50.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.310+0000 7fef7a7fc640 1 -- 172.21.15.153:0/307652229 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef64003520 con 0x7fef7c0b39f0 2024-03-27T03:05:50.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.311+0000 7fef7a7fc640 1 -- 172.21.15.153:0/307652229 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1443105107 0 0) 0x7fef64003520 con 0x7fef7c0b39f0 2024-03-27T03:05:50.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.311+0000 7fef7a7fc640 1 -- 172.21.15.153:0/307652229 >> v1:172.21.15.196:6789/0 conn(0x7fef7c0b4790 legacy=0x7fef7c0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:50.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.311+0000 7fef7a7fc640 1 -- 172.21.15.153:0/307652229 >> v1:172.21.15.153:6789/0 conn(0x7fef7c0aea90 legacy=0x7fef7c0af540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:05:50.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef7a7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:05:50.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef7a7fc640 1 monclient: found mon.noname-b 2024-03-27T03:05:50.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:50.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef7a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:05:50.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef7a7fc640 1 -- 172.21.15.153:0/307652229 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c13fd60 con 0x7fef7c0b39f0 2024-03-27T03:05:50.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef7a7fc640 10 monclient: _finish_auth 0 2024-03-27T03:05:50.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef7a7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:05:50.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:50.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef7a7fc640 1 -- 172.21.15.153:0/307652229 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fef74001a10 con 0x7fef7c0b39f0 2024-03-27T03:05:50.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:50.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef7a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef7a7fc640 10 monclient: dump: 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef81e7b640 5 monclient: authenticate success, global_id 4244 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.313+0000 7fef81e7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.314+0000 7fef7a7fc640 1 -- 172.21.15.153:0/307652229 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef74001be0 con 0x7fef7c0b39f0 2024-03-27T03:05:50.310 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.314+0000 7fef7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.314+0000 7fef7a7fc640 1 -- 172.21.15.153:0/307652229 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fef74002030 con 0x7fef7c0b39f0 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.314+0000 7fef81e7b640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.314+0000 7fef7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.314+0000 7fef7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.314+0000 7fef7a7fc640 10 monclient: dump: 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:50.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.314+0000 7fef81e7b640 4 set_mon_vals no callback set 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.315+0000 7fef81e7b640 10 monclient: shutdown 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.315+0000 7fef81e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.315+0000 7fef81e7b640 1 -- 172.21.15.153:0/307652229 >> v1:172.21.15.153:6790/0 conn(0x7fef7c0b39f0 legacy=0x7fef7c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.315+0000 7fef81e7b640 1 -- 172.21.15.153:0/307652229 shutdown_connections 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.315+0000 7fef81e7b640 1 -- 172.21.15.153:0/307652229 >> 172.21.15.153:0/307652229 conn(0x7fef7c066a00 msgr2=0x7fef7c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.315+0000 7fef81e7b640 1 -- 172.21.15.153:0/307652229 shutdown_connections 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.315+0000 7fef81e7b640 1 -- 172.21.15.153:0/307652229 wait complete. 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.316+0000 7fef81e7b640 10 monclient: build_initial_monmap 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.316+0000 7fef81e7b640 10 monclient: monmap: 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:05:50.312 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:05:50.313 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:50.313 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:50.313 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:50.313 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:50.313 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:50.313 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:50.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.317+0000 7fef81e7b640 1 librados: starting msgr at 2024-03-27T03:05:50.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.317+0000 7fef81e7b640 1 librados: starting objecter 2024-03-27T03:05:50.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.317+0000 7fef81e7b640 1 Processor -- start 2024-03-27T03:05:50.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.317+0000 7fef81e7b640 1 -- start start 2024-03-27T03:05:50.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.317+0000 7fef81e7b640 1 librados: setting wanted keys 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.317+0000 7fef81e7b640 1 librados: calling monclient init 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.317+0000 7fef81e7b640 10 monclient: init 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef81e7b640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef81e7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef81e7b640 10 monclient(hunting): picked mon.noname-b con 0x7fef7c0b39f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef81e7b640 10 monclient(hunting): picked mon.noname-c con 0x7fef7c0b4790 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef81e7b640 10 monclient(hunting): picked mon.noname-a con 0x7fef7c14bef0 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef81e7b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c14d640 con 0x7fef7c14bef0 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef81e7b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c14db90 con 0x7fef7c0b39f0 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef81e7b640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c14de70 con 0x7fef7c0b4790 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef81e7b640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef81e7b640 10 monclient(hunting): authenticate will time out at 1310.623912s 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef7affd640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fef7c0b4790 0x7fef7c14c600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:40054/0 (socket says 172.21.15.153:40054) 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef7affd640 1 -- 172.21.15.153:0/83946499 learned_addr learned my addr 172.21.15.153:0/83946499 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:05:50.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef63fff640 10 client.?.objecter ms_handle_connect 0x7fef7c0b4790 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef63fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef63fff640 10 client.?.objecter ms_handle_connect 0x7fef7c0b39f0 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef63fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142552064 0 0) 0x7fef7c14de70 con 0x7fef7c0b4790 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef63fff640 10 monclient(hunting): _negotiate 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef63fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef63fff640 10 monclient(hunting): my global_id is 4247 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.318+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef68004eb0 con 0x7fef7c0b4790 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2869734915 0 0) 0x7fef68004eb0 con 0x7fef7c0b4790 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef7c14de70 con 0x7fef7c0b4790 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fef70001960 con 0x7fef7c0b4790 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:05:50.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 10 monclient(hunting): dump: 2024-03-27T03:05:50.322 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:50.322 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:50.322 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:50.322 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:50.322 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:50.322 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:50.322 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:50.322 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:50.322 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:50.322 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146605706 0 0) 0x7fef7c14db90 con 0x7fef7c0b39f0 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 10 monclient(hunting): _negotiate 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 10 monclient(hunting): my global_id is 4231 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef68004eb0 con 0x7fef7c0b39f0 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1805811167 0 0) 0x7fef7c14de70 con 0x7fef7c0b4790 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 >> v1:172.21.15.196:6789/0 conn(0x7fef7c0b39f0 legacy=0x7fef7c12dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.319+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 >> v1:172.21.15.153:6789/0 conn(0x7fef7c14bef0 legacy=0x7fef7c1e61b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef63fff640 10 monclient: _finish_hunting 0 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef63fff640 1 monclient: found mon.c 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef63fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:05:50.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c14d8b0 con 0x7fef7c0b4790 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef63fff640 10 monclient: _finish_auth 0 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef63fff640 10 monclient: _check_auth_tickets 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef81e7b640 5 monclient: authenticate success, global_id 4247 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef81e7b640 10 monclient: _renew_subs 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef81e7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef81e7b640 1 -- 172.21.15.153:0/83946499 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7c14ceb0 con 0x7fef7c0b4790 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef81e7b640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef81e7b640 10 monclient: _renew_subs 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef81e7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.321+0000 7fef81e7b640 1 -- 172.21.15.153:0/83946499 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7c1e78d0 con 0x7fef7c0b4790 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.322+0000 7fef81e7b640 1 librados: init done 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.323+0000 7fef81e7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.323+0000 7fef81e7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:50.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.323+0000 7fef81e7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.323+0000 7fef81e7b640 1 -- 172.21.15.153:0/83946499 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c067fc0 con 0x7fef7c0b4790 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.323+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef70001c40 con 0x7fef7c0b4790 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.323+0000 7fef63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.323+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fef70001ac0 con 0x7fef7c0b4790 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.323+0000 7fef63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.323+0000 7fef63fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.323+0000 7fef63fff640 10 monclient: dump: 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:50.325 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:50.326 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:50.326 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:50.326 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:50.326 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.323+0000 7fef79ffb640 4 set_mon_vals no callback set 2024-03-27T03:05:50.326 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.325+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fef700038a0 con 0x7fef7c0b4790 2024-03-27T03:05:50.326 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.325+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fef700848c0 con 0x7fef7c0b4790 2024-03-27T03:05:50.326 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.325+0000 7fef63fff640 10 client.4247.objecter ms_dispatch 0x7fef7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:05:50.326 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.325+0000 7fef63fff640 3 client.4247.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:05:50.326 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.325+0000 7fef63fff640 3 client.4247.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:05:50.326 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.325+0000 7fef63fff640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-27T03:05:50.327 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.325+0000 7fef7b7fe640 10 monclient: get_auth_request con 0x7fef6806dcb0 auth_method 0 2024-03-27T03:05:50.327 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.326+0000 7fef63fff640 10 client.4247.objecter ms_handle_connect 0x7fef6806dcb0 2024-03-27T03:05:50.327 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.333+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef700501a0 con 0x7fef7c0b4790 2024-03-27T03:05:50.327 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.333+0000 7fef63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:50.327 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.333+0000 7fef63fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:50.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.632+0000 7fef81e7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:50.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.632+0000 7fef81e7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:50.626 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.632+0000 7fef81e7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:05:50.626 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.632+0000 7fef81e7b640 1 -- 172.21.15.153:0/83946499 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef7c075f10 con 0x7fef7c0b4790 2024-03-27T03:05:50.626 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.633+0000 7fef63fff640 1 -- 172.21.15.153:0/83946499 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fef700520d0 con 0x7fef7c0b4790 2024-03-27T03:05:50.627 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.633+0000 7fef63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:50.627 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.633+0000 7fef63fff640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:50.627 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:05:50.627 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:05:50.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.636+0000 7fef81e7b640 10 librados: watch_flush enter 2024-03-27T03:05:50.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.637+0000 7fef81e7b640 10 librados: watch_flush exit 2024-03-27T03:05:50.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.637+0000 7fef81e7b640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-27T03:05:50.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.637+0000 7fef81e7b640 10 client.4247.objecter successfully canceled tick 2024-03-27T03:05:50.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.637+0000 7fef81e7b640 1 -- 172.21.15.153:0/83946499 >> v1:172.21.15.153:6816/2637937781 conn(0x7fef6806dcb0 legacy=0x7fef68070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:50.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.637+0000 7fef81e7b640 10 monclient: shutdown 2024-03-27T03:05:50.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.637+0000 7fef81e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:50.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.637+0000 7fef81e7b640 1 -- 172.21.15.153:0/83946499 >> v1:172.21.15.153:6790/0 conn(0x7fef7c0b4790 legacy=0x7fef7c14c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:50.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.637+0000 7fef81e7b640 1 -- 172.21.15.153:0/83946499 shutdown_connections 2024-03-27T03:05:50.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.637+0000 7fef81e7b640 1 -- 172.21.15.153:0/83946499 >> 172.21.15.153:0/83946499 conn(0x7fef7c066a00 msgr2=0x7fef7c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:50.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.637+0000 7fef81e7b640 1 -- 172.21.15.153:0/83946499 shutdown_connections 2024-03-27T03:05:50.631 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.637+0000 7fef81e7b640 1 -- 172.21.15.153:0/83946499 wait complete. 2024-03-27T03:05:50.631 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:50.638+0000 7fef81e7b640 1 librados: shutdown 2024-03-27T03:05:50.646 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:05:57.454 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:05:57.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.601+0000 7fe8e2436640 10 monclient: get_monmap_and_config 2024-03-27T03:05:57.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.601+0000 7fe8e2436640 10 monclient: build_initial_monmap 2024-03-27T03:05:57.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.601+0000 7fe8e2436640 10 monclient: monmap: 2024-03-27T03:05:57.594 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:05:57.594 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:57.594 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:05:57.602459+0000 2024-03-27T03:05:57.595 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:05:57.602459+0000 2024-03-27T03:05:57.595 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:57.595 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:57.595 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:57.595 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:05:57.595 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:05:57.595 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:57.598 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.605+0000 7fe8e2436640 1 Processor -- start 2024-03-27T03:05:57.598 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.605+0000 7fe8e2436640 1 -- start start 2024-03-27T03:05:57.598 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.605+0000 7fe8e2436640 10 monclient: init 2024-03-27T03:05:57.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.606+0000 7fe8e2436640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:57.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.606+0000 7fe8e2436640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:05:57.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.606+0000 7fe8e2436640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d40389e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:57.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.606+0000 7fe8e2436640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d4038dc0 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:57.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.606+0000 7fe8e2436640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d4030660 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:57.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.606+0000 7fe8e2436640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d40c54c0 con 0x7fe8d40389e0 2024-03-27T03:05:57.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.606+0000 7fe8e2436640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d4036430 con 0x7fe8d4030660 2024-03-27T03:05:57.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.606+0000 7fe8e2436640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d4039840 con 0x7fe8d4038dc0 2024-03-27T03:05:57.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.606+0000 7fe8e2436640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:57.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.606+0000 7fe8e2436640 10 monclient(hunting): authenticate will time out at 1317.912169s 2024-03-27T03:05:57.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.607+0000 7fe8e1c35640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe8d4030660 0x7fe8d40d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:50940/0 (socket says 172.21.15.153:50940) 2024-03-27T03:05:57.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.607+0000 7fe8e1c35640 1 -- 172.21.15.153:0/1747522279 learned_addr learned my addr 172.21.15.153:0/1747522279 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:05:57.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.608+0000 7fe8d3fff640 1 -- 172.21.15.153:0/1747522279 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65050252 0 0) 0x7fe8d4036430 con 0x7fe8d4030660 2024-03-27T03:05:57.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.608+0000 7fe8d3fff640 10 monclient(hunting): _negotiate 2024-03-27T03:05:57.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.608+0000 7fe8d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:57.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.608+0000 7fe8d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:57.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.608+0000 7fe8d3fff640 10 monclient(hunting): my global_id is 4234 2024-03-27T03:05:57.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.608+0000 7fe8d3fff640 1 -- 172.21.15.153:0/1747522279 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c8003520 con 0x7fe8d4030660 2024-03-27T03:05:57.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.608+0000 7fe8d3fff640 1 -- 172.21.15.153:0/1747522279 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 556199834 0 0) 0x7fe8c8003520 con 0x7fe8d4030660 2024-03-27T03:05:57.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.608+0000 7fe8d3fff640 1 -- 172.21.15.153:0/1747522279 >> v1:172.21.15.153:6790/0 conn(0x7fe8d4038dc0 legacy=0x7fe8d40cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:05:57.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 1 -- 172.21.15.153:0/1747522279 >> v1:172.21.15.153:6789/0 conn(0x7fe8d40389e0 legacy=0x7fe8d4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 10 monclient: _finish_hunting 0 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 1 monclient: found mon.noname-c 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 1 -- 172.21.15.153:0/1747522279 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d4035690 con 0x7fe8d4030660 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 10 monclient: _finish_auth 0 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 10 monclient: _check_auth_tickets 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 1 -- 172.21.15.153:0/1747522279 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe8dc04f730 con 0x7fe8d4030660 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.610+0000 7fe8d3fff640 10 monclient: dump: 2024-03-27T03:05:57.606 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.611+0000 7fe8e2436640 5 monclient: authenticate success, global_id 4234 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.611+0000 7fe8e2436640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.611+0000 7fe8d3fff640 1 -- 172.21.15.153:0/1747522279 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8dc0008d0 con 0x7fe8d4030660 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.611+0000 7fe8d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.611+0000 7fe8d3fff640 1 -- 172.21.15.153:0/1747522279 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe8dc0758b0 con 0x7fe8d4030660 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.611+0000 7fe8d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:57.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.611+0000 7fe8d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.611+0000 7fe8d3fff640 10 monclient: dump: 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.611+0000 7fe8e2436640 10 monclient: get_monmap_and_config success 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.611+0000 7fe8e2436640 4 set_mon_vals no callback set 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.612+0000 7fe8e2436640 10 monclient: shutdown 2024-03-27T03:05:57.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.612+0000 7fe8e2436640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:57.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.612+0000 7fe8e2436640 1 -- 172.21.15.153:0/1747522279 >> v1:172.21.15.196:6789/0 conn(0x7fe8d4030660 legacy=0x7fe8d40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:57.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.612+0000 7fe8e2436640 1 -- 172.21.15.153:0/1747522279 shutdown_connections 2024-03-27T03:05:57.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.612+0000 7fe8e2436640 1 -- 172.21.15.153:0/1747522279 >> 172.21.15.153:0/1747522279 conn(0x7fe8d4005c00 msgr2=0x7fe8d4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:57.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.613+0000 7fe8e2436640 1 -- 172.21.15.153:0/1747522279 shutdown_connections 2024-03-27T03:05:57.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.613+0000 7fe8e2436640 1 -- 172.21.15.153:0/1747522279 wait complete. 2024-03-27T03:05:57.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.614+0000 7fe8e2436640 10 monclient: build_initial_monmap 2024-03-27T03:05:57.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.614+0000 7fe8e2436640 10 monclient: monmap: 2024-03-27T03:05:57.610 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:05:57.610 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:05:57.610 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:05:57.610 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:05:57.610 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:05:57.610 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:57.610 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.615+0000 7fe8e2436640 1 librados: starting msgr at 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.615+0000 7fe8e2436640 1 librados: starting objecter 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.615+0000 7fe8e2436640 1 Processor -- start 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.615+0000 7fe8e2436640 1 -- start start 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.615+0000 7fe8e2436640 1 librados: setting wanted keys 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.615+0000 7fe8e2436640 1 librados: calling monclient init 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.615+0000 7fe8e2436640 10 monclient: init 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.616+0000 7fe8e2436640 10 monclient: _reopen_session rank -1 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.616+0000 7fe8e2436640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.616+0000 7fe8e2436640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d4030660 addr v1:172.21.15.153:6789/0 2024-03-27T03:05:57.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.616+0000 7fe8e2436640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d40389e0 addr v1:172.21.15.153:6790/0 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.619+0000 7fe8e2436640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d402ef40 addr v1:172.21.15.196:6789/0 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.619+0000 7fe8e2436640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d402f580 con 0x7fe8d4030660 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.619+0000 7fe8e1434640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe8d4030660 0x7fe8d41662d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:59895/0 (socket says 172.21.15.153:59895) 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.619+0000 7fe8e1434640 1 -- 172.21.15.153:0/285962829 learned_addr learned my addr 172.21.15.153:0/285962829 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.619+0000 7fe8d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe8d4030660 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.619+0000 7fe8d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.619+0000 7fe8e2436640 1 -- 172.21.15.153:0/285962829 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d400a0b0 con 0x7fe8d402ef40 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.619+0000 7fe8e2436640 1 -- 172.21.15.153:0/285962829 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d402fba0 con 0x7fe8d40389e0 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.619+0000 7fe8e2436640 10 monclient(hunting): _renew_subs 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.619+0000 7fe8e2436640 10 monclient(hunting): authenticate will time out at 1317.925516s 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe8d402ef40 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436218155 0 0) 0x7fe8d402f580 con 0x7fe8d4030660 2024-03-27T03:05:57.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 monclient(hunting): my global_id is 4236 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c0004eb0 con 0x7fe8d4030660 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608727099 0 0) 0x7fe8d400a0b0 con 0x7fe8d402ef40 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 monclient(hunting): my global_id is 4237 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c00051d0 con 0x7fe8d402ef40 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2382277194 0 0) 0x7fe8c00051d0 con 0x7fe8d402ef40 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8d400a0b0 con 0x7fe8d402ef40 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe8dc04f400 con 0x7fe8d402ef40 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:05:57.618 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 monclient(hunting): dump: 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.620+0000 7fe8d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe8d40389e0 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3240322470 0 0) 0x7fe8d400a0b0 con 0x7fe8d402ef40 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 >> v1:172.21.15.153:6790/0 conn(0x7fe8d40389e0 legacy=0x7fe8d416c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:57.619 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 >> v1:172.21.15.153:6789/0 conn(0x7fe8d4030660 legacy=0x7fe8d41662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 1 monclient: found mon.b 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d402fd30 con 0x7fe8d402ef40 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 10 monclient: _finish_auth 0 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8dc04f730 con 0x7fe8d402ef40 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d2ffd640 4 set_mon_vals no callback set 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8e2436640 5 monclient: authenticate success, global_id 4237 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8e2436640 10 monclient: _renew_subs 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8e2436640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8e2436640 1 -- 172.21.15.153:0/285962829 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d40c79a0 con 0x7fe8d402ef40 2024-03-27T03:05:57.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8e2436640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8e2436640 10 monclient: _renew_subs 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8e2436640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8e2436640 1 -- 172.21.15.153:0/285962829 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d41669e0 con 0x7fe8d402ef40 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8e2436640 1 librados: init done 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe8dc067bf0 con 0x7fe8d402ef40 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.621+0000 7fe8d1ffb640 10 monclient: dump: 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:05:57.621 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:05:57.622 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:05:57.622 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:05:57.622 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:05:57.622 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.623+0000 7fe8e2436640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:57.622 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.623+0000 7fe8e2436640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:57.622 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.623+0000 7fe8e2436640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:57.622 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.623+0000 7fe8e2436640 1 -- 172.21.15.153:0/285962829 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d4007b70 con 0x7fe8d402ef40 2024-03-27T03:05:57.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.625+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fe8dc067dd0 con 0x7fe8d402ef40 2024-03-27T03:05:57.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.625+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fe8dc0ff410 con 0x7fe8d402ef40 2024-03-27T03:05:57.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.625+0000 7fe8d1ffb640 10 client.4237.objecter ms_dispatch 0x7fe8dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:05:57.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.625+0000 7fe8d1ffb640 3 client.4237.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:05:57.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.625+0000 7fe8d1ffb640 3 client.4237.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:05:57.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.625+0000 7fe8d1ffb640 20 client.4237.objecter dump_active .. 0 homeless 2024-03-27T03:05:57.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.626+0000 7fe8e1434640 10 monclient: get_auth_request con 0x7fe8c006de90 auth_method 0 2024-03-27T03:05:57.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.626+0000 7fe8d1ffb640 10 client.4237.objecter ms_handle_connect 0x7fe8c006de90 2024-03-27T03:05:57.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.631+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8dc0cacf0 con 0x7fe8d402ef40 2024-03-27T03:05:57.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.631+0000 7fe8d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:05:57.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.631+0000 7fe8d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:05:57.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.952+0000 7fe8e2436640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:57.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.952+0000 7fe8e2436640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:57.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.952+0000 7fe8e2436640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:05:57.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.952+0000 7fe8e2436640 1 -- 172.21.15.153:0/285962829 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8d402f380 con 0x7fe8d402ef40 2024-03-27T03:05:57.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.953+0000 7fe8d1ffb640 1 -- 172.21.15.153:0/285962829 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fe8dc0ccc20 con 0x7fe8d402ef40 2024-03-27T03:05:57.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.953+0000 7fe8d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:05:57.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.953+0000 7fe8d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:05:57.946 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:05:57.946 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:05:57.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.956+0000 7fe8e2436640 10 librados: watch_flush enter 2024-03-27T03:05:57.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.956+0000 7fe8e2436640 10 librados: watch_flush exit 2024-03-27T03:05:57.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.956+0000 7fe8e2436640 20 client.4237.objecter shutdown clearing up homeless session... 2024-03-27T03:05:57.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.956+0000 7fe8e2436640 10 client.4237.objecter successfully canceled tick 2024-03-27T03:05:57.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.956+0000 7fe8e2436640 1 -- 172.21.15.153:0/285962829 >> v1:172.21.15.153:6816/2637937781 conn(0x7fe8c006de90 legacy=0x7fe8c0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:57.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.956+0000 7fe8e2436640 10 monclient: shutdown 2024-03-27T03:05:57.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.956+0000 7fe8e2436640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:05:57.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.956+0000 7fe8e2436640 1 -- 172.21.15.153:0/285962829 >> v1:172.21.15.196:6789/0 conn(0x7fe8d402ef40 legacy=0x7fe8d4174a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:05:57.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.956+0000 7fe8e2436640 1 -- 172.21.15.153:0/285962829 shutdown_connections 2024-03-27T03:05:57.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.956+0000 7fe8e2436640 1 -- 172.21.15.153:0/285962829 >> 172.21.15.153:0/285962829 conn(0x7fe8d4005c00 msgr2=0x7fe8d403ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:05:57.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.957+0000 7fe8e2436640 1 -- 172.21.15.153:0/285962829 shutdown_connections 2024-03-27T03:05:57.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.957+0000 7fe8e2436640 1 -- 172.21.15.153:0/285962829 wait complete. 2024-03-27T03:05:57.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:05:57.957+0000 7fe8e2436640 1 librados: shutdown 2024-03-27T03:05:57.967 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:06:04.774 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:06:04.925 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.931+0000 7fd1a756d640 10 monclient: get_monmap_and_config 2024-03-27T03:06:04.925 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.931+0000 7fd1a756d640 10 monclient: build_initial_monmap 2024-03-27T03:06:04.925 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.931+0000 7fd1a756d640 10 monclient: monmap: 2024-03-27T03:06:04.925 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:04.925 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:04.926 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:06:04.932507+0000 2024-03-27T03:06:04.926 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:06:04.932507+0000 2024-03-27T03:06:04.926 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:04.926 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:04.926 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:04.926 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:06:04.926 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:06:04.926 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:04.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.932+0000 7fd1a756d640 1 Processor -- start 2024-03-27T03:06:04.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.932+0000 7fd1a756d640 1 -- start start 2024-03-27T03:06:04.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.932+0000 7fd1a756d640 10 monclient: init 2024-03-27T03:06:04.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.934+0000 7fd1a756d640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:04.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.934+0000 7fd1a756d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:06:04.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.934+0000 7fd1a756d640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a00a9750 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:04.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.934+0000 7fd1a756d640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a00ae220 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:04.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.934+0000 7fd1a756d640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a013b9d0 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:04.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.934+0000 7fd1a756d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a0142490 con 0x7fd1a013b9d0 2024-03-27T03:06:04.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.934+0000 7fd1a756d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a00ac130 con 0x7fd1a00a9750 2024-03-27T03:06:04.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.934+0000 7fd1a756d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a00ac410 con 0x7fd1a00ae220 2024-03-27T03:06:04.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.934+0000 7fd1a756d640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:04.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.934+0000 7fd1a756d640 10 monclient(hunting): authenticate will time out at 1325.239832s 2024-03-27T03:06:04.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.937+0000 7fd1a5d6a640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fd1a00ae220 0x7fd1a013b2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:34141/0 (socket says 172.21.15.153:34141) 2024-03-27T03:06:04.931 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.938+0000 7fd1a5d6a640 1 -- 172.21.15.153:0/2802338108 learned_addr learned my addr 172.21.15.153:0/2802338108 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447876627 0 0) 0x7fd1a00ac130 con 0x7fd1a00a9750 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient(hunting): _negotiate 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient(hunting): my global_id is 4240 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd18c003520 con 0x7fd1a00a9750 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636405565 0 0) 0x7fd1a00ac410 con 0x7fd1a00ae220 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient(hunting): _negotiate 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient(hunting): my global_id is 4253 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a00ac130 con 0x7fd1a00ae220 2024-03-27T03:06:04.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145384233 0 0) 0x7fd1a0142490 con 0x7fd1a013b9d0 2024-03-27T03:06:04.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient(hunting): _negotiate 2024-03-27T03:06:04.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient(hunting): my global_id is 4239 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a00ac410 con 0x7fd1a013b9d0 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1501118332 0 0) 0x7fd18c003520 con 0x7fd1a00a9750 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 >> v1:172.21.15.153:6790/0 conn(0x7fd1a00ae220 legacy=0x7fd1a013b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 >> v1:172.21.15.153:6789/0 conn(0x7fd1a013b9d0 legacy=0x7fd1a0149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient: _finish_hunting 0 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 1 monclient: found mon.noname-c 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.941+0000 7fd1a5569640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.942+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a00afa40 con 0x7fd1a00a9750 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.942+0000 7fd1a5569640 10 monclient: _finish_auth 0 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.942+0000 7fd1a5569640 10 monclient: _check_auth_tickets 2024-03-27T03:06:04.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.942+0000 7fd1a5569640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.942+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd198001a10 con 0x7fd1a00a9750 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.942+0000 7fd1a5569640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.942+0000 7fd1a5569640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.942+0000 7fd1a5569640 10 monclient: dump: 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:04.939 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.945+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd198001c30 con 0x7fd1a00a9750 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.945+0000 7fd1a5569640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.945+0000 7fd1a5569640 1 -- 172.21.15.153:0/2802338108 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd198001f40 con 0x7fd1a00a9750 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.945+0000 7fd1a756d640 5 monclient: authenticate success, global_id 4240 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.945+0000 7fd1a756d640 10 monclient: get_monmap_and_config success 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.945+0000 7fd1a756d640 4 set_mon_vals no callback set 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.945+0000 7fd1a5569640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.946+0000 7fd1a5569640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.946+0000 7fd1a5569640 10 monclient: dump: 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:04.941 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:04.942 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:04.942 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:04.942 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:04.942 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:04.942 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:04.942 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:04.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.946+0000 7fd1a756d640 10 monclient: shutdown 2024-03-27T03:06:04.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.946+0000 7fd1a756d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:04.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.946+0000 7fd1a756d640 1 -- 172.21.15.153:0/2802338108 >> v1:172.21.15.196:6789/0 conn(0x7fd1a00a9750 legacy=0x7fd1a0141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:04.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.947+0000 7fd1a756d640 1 -- 172.21.15.153:0/2802338108 shutdown_connections 2024-03-27T03:06:04.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.947+0000 7fd1a756d640 1 -- 172.21.15.153:0/2802338108 >> 172.21.15.153:0/2802338108 conn(0x7fd1a00673e0 msgr2=0x7fd1a00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:04.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.950+0000 7fd1a756d640 1 -- 172.21.15.153:0/2802338108 shutdown_connections 2024-03-27T03:06:04.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.950+0000 7fd1a756d640 1 -- 172.21.15.153:0/2802338108 wait complete. 2024-03-27T03:06:04.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.950+0000 7fd1a756d640 10 monclient: build_initial_monmap 2024-03-27T03:06:04.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.950+0000 7fd1a756d640 10 monclient: monmap: 2024-03-27T03:06:04.944 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:04.944 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:06:04.944 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:06:04.944 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:06:04.944 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:04.944 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:04.944 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:04.944 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:06:04.944 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:06:04.944 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:04.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.951+0000 7fd1a756d640 1 librados: starting msgr at 2024-03-27T03:06:04.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.951+0000 7fd1a756d640 1 librados: starting objecter 2024-03-27T03:06:04.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.951+0000 7fd1a756d640 1 Processor -- start 2024-03-27T03:06:04.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.951+0000 7fd1a756d640 1 -- start start 2024-03-27T03:06:04.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.951+0000 7fd1a756d640 1 librados: setting wanted keys 2024-03-27T03:06:04.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.951+0000 7fd1a756d640 1 librados: calling monclient init 2024-03-27T03:06:04.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.951+0000 7fd1a756d640 10 monclient: init 2024-03-27T03:06:04.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.952+0000 7fd1a756d640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:04.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.952+0000 7fd1a756d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:06:04.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.953+0000 7fd1a756d640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a00a9750 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:04.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.953+0000 7fd1a756d640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a00ae220 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:04.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.953+0000 7fd1a756d640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a013b9d0 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:04.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.955+0000 7fd1a756d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a0071a10 con 0x7fd1a00ae220 2024-03-27T03:06:04.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd1a6d6c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd1a013b9d0 0x7fd1a01dd8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:36593/0 (socket says 172.21.15.153:36593) 2024-03-27T03:06:04.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd1a6d6c640 1 -- 172.21.15.153:0/3710731177 learned_addr learned my addr 172.21.15.153:0/3710731177 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:04.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd1a756d640 1 -- 172.21.15.153:0/3710731177 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a0072600 con 0x7fd1a013b9d0 2024-03-27T03:06:04.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd1a756d640 1 -- 172.21.15.153:0/3710731177 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a006f800 con 0x7fd1a00a9750 2024-03-27T03:06:04.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd1a756d640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:04.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd1a756d640 10 monclient(hunting): authenticate will time out at 1325.262057s 2024-03-27T03:06:04.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd196ffd640 10 client.?.objecter ms_handle_connect 0x7fd1a013b9d0 2024-03-27T03:06:04.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd196ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:04.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579793016 0 0) 0x7fd1a0072600 con 0x7fd1a013b9d0 2024-03-27T03:06:04.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd196ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:06:04.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd196ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:04.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd196ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:04.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd196ffd640 10 monclient(hunting): my global_id is 4243 2024-03-27T03:06:04.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.956+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd184004eb0 con 0x7fd1a013b9d0 2024-03-27T03:06:04.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.959+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1385778420 0 0) 0x7fd184004eb0 con 0x7fd1a013b9d0 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.959+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1a0072600 con 0x7fd1a013b9d0 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.959+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd19c003f50 con 0x7fd1a013b9d0 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.959+0000 7fd196ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.959+0000 7fd196ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.959+0000 7fd196ffd640 10 monclient(hunting): dump: 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:04.955 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.959+0000 7fd196ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 10 client.?.objecter ms_handle_connect 0x7fd1a00a9750 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831792530 0 0) 0x7fd1a006f800 con 0x7fd1a00a9750 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 10 monclient(hunting): my global_id is 4256 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd184004eb0 con 0x7fd1a00a9750 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1132848171 0 0) 0x7fd1a0072600 con 0x7fd1a013b9d0 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 >> v1:172.21.15.153:6790/0 conn(0x7fd1a00a9750 legacy=0x7fd1a0071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 >> v1:172.21.15.153:6789/0 conn(0x7fd1a00ae220 legacy=0x7fd1a006fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:06:04.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 1 monclient: found mon.b 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a006f990 con 0x7fd1a013b9d0 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 10 monclient: _finish_auth 0 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd196ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd1a756d640 5 monclient: authenticate success, global_id 4243 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd1a756d640 10 monclient: _renew_subs 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd1a756d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd1a756d640 1 -- 172.21.15.153:0/3710731177 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a0070530 con 0x7fd1a013b9d0 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd1a756d640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd1a756d640 10 monclient: _renew_subs 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd1a756d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:04.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd1a756d640 1 -- 172.21.15.153:0/3710731177 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a00b2cd0 con 0x7fd1a013b9d0 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.960+0000 7fd1a756d640 1 librados: init done 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.961+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd19c0041b0 con 0x7fd1a013b9d0 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.961+0000 7fd196ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.961+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd19c001180 con 0x7fd1a013b9d0 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.961+0000 7fd196ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.961+0000 7fd196ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.961+0000 7fd1a4d68640 4 set_mon_vals no callback set 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.961+0000 7fd196ffd640 10 monclient: dump: 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:04.958 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:04.959 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:04.959 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:04.959 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:04.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.962+0000 7fd1a756d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:04.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.962+0000 7fd1a756d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:04.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.962+0000 7fd1a756d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:04.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.962+0000 7fd1a756d640 1 -- 172.21.15.153:0/3710731177 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd188005080 con 0x7fd1a013b9d0 2024-03-27T03:06:04.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.964+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fd19c001360 con 0x7fd1a013b9d0 2024-03-27T03:06:04.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.965+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fd19c084920 con 0x7fd1a013b9d0 2024-03-27T03:06:04.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.965+0000 7fd196ffd640 10 client.4243.objecter ms_dispatch 0x7fd1a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:06:04.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.965+0000 7fd196ffd640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:06:04.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.965+0000 7fd196ffd640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:06:04.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.965+0000 7fd196ffd640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-27T03:06:04.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.965+0000 7fd1a656b640 10 monclient: get_auth_request con 0x7fd18406dea0 auth_method 0 2024-03-27T03:06:04.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.966+0000 7fd196ffd640 10 client.4243.objecter ms_handle_connect 0x7fd18406dea0 2024-03-27T03:06:04.964 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.971+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd19c050200 con 0x7fd1a013b9d0 2024-03-27T03:06:04.965 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.971+0000 7fd196ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:04.965 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:04.971+0000 7fd196ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:06:05.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.293+0000 7fd1a756d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:05.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.293+0000 7fd1a756d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:05.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.293+0000 7fd1a756d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:05.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.293+0000 7fd1a756d640 1 -- 172.21.15.153:0/3710731177 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd188005330 con 0x7fd1a013b9d0 2024-03-27T03:06:05.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.297+0000 7fd196ffd640 1 -- 172.21.15.153:0/3710731177 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fd19c050200 con 0x7fd1a013b9d0 2024-03-27T03:06:05.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.297+0000 7fd196ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:05.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.297+0000 7fd196ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:06:05.291 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:06:05.291 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:06:05.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.300+0000 7fd1a756d640 10 librados: watch_flush enter 2024-03-27T03:06:05.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.301+0000 7fd1a756d640 10 librados: watch_flush exit 2024-03-27T03:06:05.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.301+0000 7fd1a756d640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-27T03:06:05.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.301+0000 7fd1a756d640 10 client.4243.objecter successfully canceled tick 2024-03-27T03:06:05.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.301+0000 7fd1a756d640 1 -- 172.21.15.153:0/3710731177 >> v1:172.21.15.153:6816/2637937781 conn(0x7fd18406dea0 legacy=0x7fd184070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:05.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.301+0000 7fd1a756d640 10 monclient: shutdown 2024-03-27T03:06:05.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.301+0000 7fd1a756d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:05.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.301+0000 7fd1a756d640 1 -- 172.21.15.153:0/3710731177 >> v1:172.21.15.196:6789/0 conn(0x7fd1a013b9d0 legacy=0x7fd1a01dd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:05.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.301+0000 7fd1a756d640 1 -- 172.21.15.153:0/3710731177 shutdown_connections 2024-03-27T03:06:05.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.301+0000 7fd1a756d640 1 -- 172.21.15.153:0/3710731177 >> 172.21.15.153:0/3710731177 conn(0x7fd1a00673e0 msgr2=0x7fd1a0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:05.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.301+0000 7fd1a756d640 1 -- 172.21.15.153:0/3710731177 shutdown_connections 2024-03-27T03:06:05.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.302+0000 7fd1a756d640 1 -- 172.21.15.153:0/3710731177 wait complete. 2024-03-27T03:06:05.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:05.302+0000 7fd1a756d640 1 librados: shutdown 2024-03-27T03:06:05.312 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:06:12.122 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:06:12.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.280+0000 7fa1816b5640 10 monclient: get_monmap_and_config 2024-03-27T03:06:12.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.280+0000 7fa1816b5640 10 monclient: build_initial_monmap 2024-03-27T03:06:12.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.280+0000 7fa1816b5640 10 monclient: monmap: 2024-03-27T03:06:12.275 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:12.275 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:12.275 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:06:12.282498+0000 2024-03-27T03:06:12.275 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:06:12.282498+0000 2024-03-27T03:06:12.275 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:12.275 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:12.275 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:12.275 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:06:12.275 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:06:12.275 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:12.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.281+0000 7fa1816b5640 1 Processor -- start 2024-03-27T03:06:12.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.281+0000 7fa1816b5640 1 -- start start 2024-03-27T03:06:12.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.281+0000 7fa1816b5640 10 monclient: init 2024-03-27T03:06:12.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.282+0000 7fa1816b5640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:12.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.282+0000 7fa1816b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:06:12.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.282+0000 7fa1816b5640 10 monclient(hunting): picked mon.noname-c con 0x7fa17c1457f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:12.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.282+0000 7fa1816b5640 10 monclient(hunting): picked mon.noname-a con 0x7fa17c13f650 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:12.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.282+0000 7fa1816b5640 10 monclient(hunting): picked mon.noname-b con 0x7fa17c0a9d70 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:12.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.282+0000 7fa1816b5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c0aaef0 con 0x7fa17c13f650 2024-03-27T03:06:12.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.282+0000 7fa1816b5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c145bd0 con 0x7fa17c1457f0 2024-03-27T03:06:12.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.282+0000 7fa1816b5640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c145eb0 con 0x7fa17c0a9d70 2024-03-27T03:06:12.281 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.282+0000 7fa1816b5640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:12.281 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.282+0000 7fa1816b5640 10 monclient(hunting): authenticate will time out at 1332.589004s 2024-03-27T03:06:12.281 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.286+0000 7fa17b7fe640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fa17c0a9d70 0x7fa17c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:50683/0 (socket says 172.21.15.153:50683) 2024-03-27T03:06:12.281 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.286+0000 7fa17b7fe640 1 -- 172.21.15.153:0/716039594 learned_addr learned my addr 172.21.15.153:0/716039594 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:12.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.286+0000 7fa179ffb640 1 -- 172.21.15.153:0/716039594 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537552940 0 0) 0x7fa17c145eb0 con 0x7fa17c0a9d70 2024-03-27T03:06:12.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.286+0000 7fa179ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:06:12.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.286+0000 7fa179ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:12.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.286+0000 7fa179ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.286+0000 7fa179ffb640 10 monclient(hunting): my global_id is 4259 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.286+0000 7fa179ffb640 1 -- 172.21.15.153:0/716039594 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa164003520 con 0x7fa17c0a9d70 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.286+0000 7fa179ffb640 1 -- 172.21.15.153:0/716039594 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751192755 0 0) 0x7fa17c0aaef0 con 0x7fa17c13f650 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 10 monclient(hunting): my global_id is 4242 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 1 -- 172.21.15.153:0/716039594 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1640048a0 con 0x7fa17c13f650 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 1 -- 172.21.15.153:0/716039594 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2994934311 0 0) 0x7fa1640048a0 con 0x7fa17c13f650 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 1 -- 172.21.15.153:0/716039594 >> v1:172.21.15.153:6790/0 conn(0x7fa17c0a9d70 legacy=0x7fa17c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 1 -- 172.21.15.153:0/716039594 >> v1:172.21.15.196:6789/0 conn(0x7fa17c1457f0 legacy=0x7fa17c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 1 monclient: found mon.noname-a 2024-03-27T03:06:12.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 1 -- 172.21.15.153:0/716039594 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c146340 con 0x7fa17c13f650 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 10 monclient: _finish_auth 0 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 1 -- 172.21.15.153:0/716039594 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa1740019f0 con 0x7fa17c13f650 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa1816b5640 5 monclient: authenticate success, global_id 4242 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa1816b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa179ffb640 10 monclient: dump: 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:12.287 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.287+0000 7fa1816b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.288+0000 7fa179ffb640 1 -- 172.21.15.153:0/716039594 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa174001b50 con 0x7fa17c13f650 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.288+0000 7fa179ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.288+0000 7fa179ffb640 1 -- 172.21.15.153:0/716039594 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa174001f60 con 0x7fa17c13f650 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.288+0000 7fa179ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.288+0000 7fa179ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.288+0000 7fa179ffb640 10 monclient: dump: 2024-03-27T03:06:12.288 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.289+0000 7fa1816b5640 10 monclient: get_monmap_and_config success 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.289+0000 7fa1816b5640 4 set_mon_vals no callback set 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.290+0000 7fa1816b5640 10 monclient: shutdown 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.290+0000 7fa1816b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:12.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.290+0000 7fa1816b5640 1 -- 172.21.15.153:0/716039594 >> v1:172.21.15.153:6789/0 conn(0x7fa17c13f650 legacy=0x7fa17c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:12.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.293+0000 7fa1816b5640 1 -- 172.21.15.153:0/716039594 shutdown_connections 2024-03-27T03:06:12.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.293+0000 7fa1816b5640 1 -- 172.21.15.153:0/716039594 >> 172.21.15.153:0/716039594 conn(0x7fa17c066a00 msgr2=0x7fa17c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:12.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.299+0000 7fa1816b5640 1 -- 172.21.15.153:0/716039594 shutdown_connections 2024-03-27T03:06:12.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.300+0000 7fa1816b5640 1 -- 172.21.15.153:0/716039594 wait complete. 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.300+0000 7fa1816b5640 10 monclient: build_initial_monmap 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.300+0000 7fa1816b5640 10 monclient: monmap: 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.301+0000 7fa1816b5640 1 librados: starting msgr at 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.301+0000 7fa1816b5640 1 librados: starting objecter 2024-03-27T03:06:12.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.301+0000 7fa1816b5640 1 Processor -- start 2024-03-27T03:06:12.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.302+0000 7fa1816b5640 1 -- start start 2024-03-27T03:06:12.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.302+0000 7fa1816b5640 1 librados: setting wanted keys 2024-03-27T03:06:12.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.302+0000 7fa1816b5640 1 librados: calling monclient init 2024-03-27T03:06:12.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.302+0000 7fa1816b5640 10 monclient: init 2024-03-27T03:06:12.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.303+0000 7fa1816b5640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:12.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.303+0000 7fa1816b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:06:12.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.303+0000 7fa1816b5640 10 monclient(hunting): picked mon.noname-b con 0x7fa17c0a9d70 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:12.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.303+0000 7fa1816b5640 10 monclient(hunting): picked mon.noname-a con 0x7fa17c13f650 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:12.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.303+0000 7fa1816b5640 10 monclient(hunting): picked mon.noname-c con 0x7fa17c1457f0 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:12.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.303+0000 7fa1816b5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c1e1900 con 0x7fa17c13f650 2024-03-27T03:06:12.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.303+0000 7fa1816b5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c06f920 con 0x7fa17c0a9d70 2024-03-27T03:06:12.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.303+0000 7fa1816b5640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c06eb50 con 0x7fa17c1457f0 2024-03-27T03:06:12.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.303+0000 7fa1816b5640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:12.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.303+0000 7fa1816b5640 10 monclient(hunting): authenticate will time out at 1332.609785s 2024-03-27T03:06:12.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.306+0000 7fa17affd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fa17c0a9d70 0x7fa17c1e74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:52165/0 (socket says 172.21.15.153:52165) 2024-03-27T03:06:12.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.306+0000 7fa17affd640 1 -- 172.21.15.153:0/3425534593 learned_addr learned my addr 172.21.15.153:0/3425534593 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:12.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.306+0000 7fa1637fe640 10 client.?.objecter ms_handle_connect 0x7fa17c0a9d70 2024-03-27T03:06:12.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.306+0000 7fa1637fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:12.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398179189 0 0) 0x7fa17c06f920 con 0x7fa17c0a9d70 2024-03-27T03:06:12.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 10 monclient(hunting): _negotiate 2024-03-27T03:06:12.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:12.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:12.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 10 monclient(hunting): my global_id is 4246 2024-03-27T03:06:12.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa168004eb0 con 0x7fa17c0a9d70 2024-03-27T03:06:12.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 10 client.?.objecter ms_handle_connect 0x7fa17c1457f0 2024-03-27T03:06:12.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445150151 0 0) 0x7fa17c06eb50 con 0x7fa17c1457f0 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 10 monclient(hunting): _negotiate 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 10 monclient(hunting): my global_id is 4262 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c06f920 con 0x7fa17c1457f0 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2153832062 0 0) 0x7fa168004eb0 con 0x7fa17c0a9d70 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.307+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa17c06eb50 con 0x7fa17c0a9d70 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.308+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa16c0017d0 con 0x7fa17c0a9d70 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.308+0000 7fa1637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.308+0000 7fa1637fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.308+0000 7fa1637fe640 10 monclient(hunting): dump: 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:12.304 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:12.305 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:12.305 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:12.305 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:12.305 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:12.305 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:12.305 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:12.305 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:12.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.308+0000 7fa1637fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:06:12.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.308+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 51396709 0 0) 0x7fa17c06eb50 con 0x7fa17c0a9d70 2024-03-27T03:06:12.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.308+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 >> v1:172.21.15.153:6790/0 conn(0x7fa17c1457f0 legacy=0x7fa17c1e11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:12.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.308+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 >> v1:172.21.15.153:6789/0 conn(0x7fa17c13f650 legacy=0x7fa17c1e0ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:06:12.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1637fe640 10 monclient: _finish_hunting 0 2024-03-27T03:06:12.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1637fe640 1 monclient: found mon.b 2024-03-27T03:06:12.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:12.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1637fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:12.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c138cc0 con 0x7fa17c0a9d70 2024-03-27T03:06:12.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1637fe640 10 monclient: _finish_auth 0 2024-03-27T03:06:12.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1637fe640 10 monclient: _check_auth_tickets 2024-03-27T03:06:12.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:12.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1816b5640 5 monclient: authenticate success, global_id 4246 2024-03-27T03:06:12.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1816b5640 10 monclient: _renew_subs 2024-03-27T03:06:12.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1816b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1816b5640 1 -- 172.21.15.153:0/3425534593 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17c13bb90 con 0x7fa17c0a9d70 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1816b5640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1816b5640 10 monclient: _renew_subs 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1816b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1816b5640 1 -- 172.21.15.153:0/3425534593 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17c06e700 con 0x7fa17c0a9d70 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.310+0000 7fa1816b5640 1 librados: init done 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.311+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa16c001930 con 0x7fa17c0a9d70 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.311+0000 7fa1637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.311+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa16c004a00 con 0x7fa17c0a9d70 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.311+0000 7fa1637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.311+0000 7fa1637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.311+0000 7fa1637fe640 10 monclient: dump: 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:12.307 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:12.308 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:12.308 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:12.308 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:12.308 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:12.308 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:12.308 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:12.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.311+0000 7fa178ff9640 4 set_mon_vals no callback set 2024-03-27T03:06:12.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.313+0000 7fa1816b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:12.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.313+0000 7fa1816b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:12.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.313+0000 7fa1816b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:12.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.313+0000 7fa1816b5640 1 -- 172.21.15.153:0/3425534593 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15c005080 con 0x7fa17c0a9d70 2024-03-27T03:06:12.310 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.316+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fa16c004b60 con 0x7fa17c0a9d70 2024-03-27T03:06:12.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.317+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fa16c084a00 con 0x7fa17c0a9d70 2024-03-27T03:06:12.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.317+0000 7fa1637fe640 10 client.4246.objecter ms_dispatch 0x7fa17c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:06:12.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.317+0000 7fa1637fe640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:06:12.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.317+0000 7fa1637fe640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:06:12.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.317+0000 7fa17a7fc640 10 monclient: get_auth_request con 0x7fa16806dd30 auth_method 0 2024-03-27T03:06:12.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.317+0000 7fa1637fe640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-27T03:06:12.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.318+0000 7fa1637fe640 10 client.4246.objecter ms_handle_connect 0x7fa16806dd30 2024-03-27T03:06:12.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.322+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa16c0502e0 con 0x7fa17c0a9d70 2024-03-27T03:06:12.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.322+0000 7fa1637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:12.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.322+0000 7fa1637fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:06:12.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.650+0000 7fa1816b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:12.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.650+0000 7fa1816b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:12.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.651+0000 7fa1816b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:12.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.651+0000 7fa1816b5640 1 -- 172.21.15.153:0/3425534593 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa15c005330 con 0x7fa17c0a9d70 2024-03-27T03:06:12.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.652+0000 7fa1637fe640 1 -- 172.21.15.153:0/3425534593 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fa16c052210 con 0x7fa17c0a9d70 2024-03-27T03:06:12.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.652+0000 7fa1637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:12.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.652+0000 7fa1637fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:06:12.646 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:06:12.646 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:06:12.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.655+0000 7fa1816b5640 10 librados: watch_flush enter 2024-03-27T03:06:12.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.656+0000 7fa1816b5640 10 librados: watch_flush exit 2024-03-27T03:06:12.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.656+0000 7fa1816b5640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-27T03:06:12.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.656+0000 7fa1816b5640 10 client.4246.objecter successfully canceled tick 2024-03-27T03:06:12.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.656+0000 7fa1816b5640 1 -- 172.21.15.153:0/3425534593 >> v1:172.21.15.153:6816/2637937781 conn(0x7fa16806dd30 legacy=0x7fa1680701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:12.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.656+0000 7fa1816b5640 10 monclient: shutdown 2024-03-27T03:06:12.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.656+0000 7fa1816b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:12.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.656+0000 7fa1816b5640 1 -- 172.21.15.153:0/3425534593 >> v1:172.21.15.196:6789/0 conn(0x7fa17c0a9d70 legacy=0x7fa17c1e74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:12.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.656+0000 7fa1816b5640 1 -- 172.21.15.153:0/3425534593 shutdown_connections 2024-03-27T03:06:12.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.656+0000 7fa1816b5640 1 -- 172.21.15.153:0/3425534593 >> 172.21.15.153:0/3425534593 conn(0x7fa17c066a00 msgr2=0x7fa17c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:12.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.656+0000 7fa1816b5640 1 -- 172.21.15.153:0/3425534593 shutdown_connections 2024-03-27T03:06:12.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.656+0000 7fa1816b5640 1 -- 172.21.15.153:0/3425534593 wait complete. 2024-03-27T03:06:12.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:12.657+0000 7fa1816b5640 1 librados: shutdown 2024-03-27T03:06:12.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:06:19.478 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:06:19.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.641+0000 7f89f0160640 10 monclient: get_monmap_and_config 2024-03-27T03:06:19.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.641+0000 7f89f0160640 10 monclient: build_initial_monmap 2024-03-27T03:06:19.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.641+0000 7f89f0160640 10 monclient: monmap: 2024-03-27T03:06:19.635 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:19.635 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:19.635 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:06:19.642473+0000 2024-03-27T03:06:19.635 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:06:19.642473+0000 2024-03-27T03:06:19.635 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:19.635 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:19.636 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:19.636 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:06:19.636 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:06:19.636 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:19.636 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.642+0000 7f89f0160640 1 Processor -- start 2024-03-27T03:06:19.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.644+0000 7f89f0160640 1 -- start start 2024-03-27T03:06:19.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.644+0000 7f89f0160640 10 monclient: init 2024-03-27T03:06:19.641 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.645+0000 7f89f0160640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:19.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.645+0000 7f89f0160640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:06:19.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.645+0000 7f89f0160640 10 monclient(hunting): picked mon.noname-a con 0x7f89e80ae290 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:19.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.645+0000 7f89f0160640 10 monclient(hunting): picked mon.noname-c con 0x7f89e80ad620 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:19.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.645+0000 7f89f0160640 10 monclient(hunting): picked mon.noname-b con 0x7f89e813e4f0 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:19.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.645+0000 7f89f0160640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80ac160 con 0x7f89e80ae290 2024-03-27T03:06:19.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.645+0000 7f89f0160640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80ac3d0 con 0x7f89e80ad620 2024-03-27T03:06:19.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.645+0000 7f89f0160640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80ac6b0 con 0x7f89e813e4f0 2024-03-27T03:06:19.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.645+0000 7f89f0160640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:19.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.645+0000 7f89f0160640 10 monclient(hunting): authenticate will time out at 1339.951462s 2024-03-27T03:06:19.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.648+0000 7f89ee6fd640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f89e813e4f0 0x7f89e814b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:43381/0 (socket says 172.21.15.153:43381) 2024-03-27T03:06:19.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.648+0000 7f89ee6fd640 1 -- 172.21.15.153:0/1341093370 learned_addr learned my addr 172.21.15.153:0/1341093370 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:19.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.649+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996246665 0 0) 0x7f89e80ac6b0 con 0x7f89e813e4f0 2024-03-27T03:06:19.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.649+0000 7f89ecefa640 10 monclient(hunting): _negotiate 2024-03-27T03:06:19.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.649+0000 7f89ecefa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:19.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.649+0000 7f89ecefa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:19.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.650+0000 7f89ecefa640 10 monclient(hunting): my global_id is 4265 2024-03-27T03:06:19.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.650+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d4003520 con 0x7f89e813e4f0 2024-03-27T03:06:19.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.650+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539929820 0 0) 0x7f89e80ac160 con 0x7f89e80ae290 2024-03-27T03:06:19.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.650+0000 7f89ecefa640 10 monclient(hunting): _negotiate 2024-03-27T03:06:19.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:19.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:19.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient(hunting): my global_id is 4245 2024-03-27T03:06:19.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e80ac6b0 con 0x7f89e80ae290 2024-03-27T03:06:19.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840357270 0 0) 0x7f89e80ac3d0 con 0x7f89e80ad620 2024-03-27T03:06:19.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient(hunting): _negotiate 2024-03-27T03:06:19.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:19.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:19.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient(hunting): my global_id is 4249 2024-03-27T03:06:19.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e80ac160 con 0x7f89e80ad620 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1369130044 0 0) 0x7f89d4003520 con 0x7f89e813e4f0 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 >> v1:172.21.15.196:6789/0 conn(0x7f89e80ad620 legacy=0x7f89e813dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 >> v1:172.21.15.153:6789/0 conn(0x7f89e80ae290 legacy=0x7f89e80ab290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient: _finish_hunting 0 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 1 monclient: found mon.noname-b 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e80ad100 con 0x7f89e813e4f0 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient: _finish_auth 0 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient: _check_auth_tickets 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f89e4001540 con 0x7f89e813e4f0 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.654+0000 7f89ecefa640 10 monclient: dump: 2024-03-27T03:06:19.650 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.655+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89e40016a0 con 0x7f89e813e4f0 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.655+0000 7f89ecefa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.655+0000 7f89ecefa640 1 -- 172.21.15.153:0/1341093370 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f89e4001af0 con 0x7f89e813e4f0 2024-03-27T03:06:19.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.655+0000 7f89ecefa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:19.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.655+0000 7f89ecefa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:06:19.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.655+0000 7f89ecefa640 10 monclient: dump: 2024-03-27T03:06:19.652 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:19.652 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:19.652 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:19.652 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:19.652 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:19.652 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:19.652 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:19.652 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:19.652 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:19.653 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:19.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.657+0000 7f89f0160640 5 monclient: authenticate success, global_id 4265 2024-03-27T03:06:19.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.657+0000 7f89f0160640 10 monclient: get_monmap_and_config success 2024-03-27T03:06:19.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.657+0000 7f89f0160640 4 set_mon_vals no callback set 2024-03-27T03:06:19.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.658+0000 7f89f0160640 10 monclient: shutdown 2024-03-27T03:06:19.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.658+0000 7f89f0160640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:19.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.658+0000 7f89f0160640 1 -- 172.21.15.153:0/1341093370 >> v1:172.21.15.153:6790/0 conn(0x7f89e813e4f0 legacy=0x7f89e814b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:19.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.660+0000 7f89f0160640 1 -- 172.21.15.153:0/1341093370 shutdown_connections 2024-03-27T03:06:19.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.660+0000 7f89f0160640 1 -- 172.21.15.153:0/1341093370 >> 172.21.15.153:0/1341093370 conn(0x7f89e80b3c80 msgr2=0x7f89e80b4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:19.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.660+0000 7f89f0160640 1 -- 172.21.15.153:0/1341093370 shutdown_connections 2024-03-27T03:06:19.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.661+0000 7f89f0160640 1 -- 172.21.15.153:0/1341093370 wait complete. 2024-03-27T03:06:19.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.662+0000 7f89f0160640 10 monclient: build_initial_monmap 2024-03-27T03:06:19.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.662+0000 7f89f0160640 10 monclient: monmap: 2024-03-27T03:06:19.658 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:19.658 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:06:19.658 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:06:19.658 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:06:19.658 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:19.658 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:19.658 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:19.658 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:06:19.658 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.663+0000 7f89f0160640 1 librados: starting msgr at 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.663+0000 7f89f0160640 1 librados: starting objecter 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.663+0000 7f89f0160640 1 Processor -- start 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.663+0000 7f89f0160640 1 -- start start 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.663+0000 7f89f0160640 1 librados: setting wanted keys 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.663+0000 7f89f0160640 1 librados: calling monclient init 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.663+0000 7f89f0160640 10 monclient: init 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.664+0000 7f89f0160640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.664+0000 7f89f0160640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.664+0000 7f89f0160640 10 monclient(hunting): picked mon.noname-c con 0x7f89e80ad620 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.664+0000 7f89f0160640 10 monclient(hunting): picked mon.noname-b con 0x7f89e80ae290 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.664+0000 7f89f0160640 10 monclient(hunting): picked mon.noname-a con 0x7f89e813e4f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.664+0000 7f89f0160640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e81421b0 con 0x7f89e813e4f0 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.664+0000 7f89f0160640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e806d850 con 0x7f89e80ae290 2024-03-27T03:06:19.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.664+0000 7f89f0160640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e8137ad0 con 0x7f89e80ad620 2024-03-27T03:06:19.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.664+0000 7f89f0160640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:19.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.664+0000 7f89f0160640 10 monclient(hunting): authenticate will time out at 1339.969765s 2024-03-27T03:06:19.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.672+0000 7f89ed6fb640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f89e80ae290 0x7f89e8140d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:49619/0 (socket says 172.21.15.153:49619) 2024-03-27T03:06:19.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.672+0000 7f89ed6fb640 1 -- 172.21.15.153:0/3052137472 learned_addr learned my addr 172.21.15.153:0/3052137472 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:19.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.672+0000 7f89de7fc640 10 client.?.objecter ms_handle_connect 0x7f89e80ae290 2024-03-27T03:06:19.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.672+0000 7f89de7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:19.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.673+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940863924 0 0) 0x7f89e806d850 con 0x7f89e80ae290 2024-03-27T03:06:19.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.673+0000 7f89de7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:06:19.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.673+0000 7f89de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:19.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.673+0000 7f89de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:19.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.673+0000 7f89de7fc640 10 monclient(hunting): my global_id is 4252 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.673+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89cc004eb0 con 0x7f89e80ae290 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.674+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1566803977 0 0) 0x7f89cc004eb0 con 0x7f89e80ae290 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.674+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89e806d850 con 0x7f89e80ae290 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.674+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f89d8001930 con 0x7f89e80ae290 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.674+0000 7f89de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.674+0000 7f89de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.674+0000 7f89de7fc640 10 monclient(hunting): dump: 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:19.668 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:19.669 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:19.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.674+0000 7f89de7fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:06:19.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.674+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1748744960 0 0) 0x7f89e806d850 con 0x7f89e80ae290 2024-03-27T03:06:19.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.674+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 >> v1:172.21.15.153:6790/0 conn(0x7f89e80ad620 legacy=0x7f89e8140620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:19.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.674+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 >> v1:172.21.15.153:6789/0 conn(0x7f89e813e4f0 legacy=0x7f89e81dd990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:06:19.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89de7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89de7fc640 1 monclient: found mon.b 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89de7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e8137640 con 0x7f89e80ae290 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89de7fc640 10 monclient: _finish_auth 0 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89de7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89f0160640 5 monclient: authenticate success, global_id 4252 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89f0160640 10 monclient: _renew_subs 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89f0160640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89f0160640 1 -- 172.21.15.153:0/3052137472 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e8137890 con 0x7f89e80ae290 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89f0160640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89f0160640 10 monclient: _renew_subs 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89f0160640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:19.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89f0160640 1 -- 172.21.15.153:0/3052137472 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e8135eb0 con 0x7f89e80ae290 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.675+0000 7f89f0160640 1 librados: init done 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.676+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89d8003610 con 0x7f89e80ae290 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.676+0000 7f89de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.676+0000 7f89df7fe640 4 set_mon_vals no callback set 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.676+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f89d8003840 con 0x7f89e80ae290 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.676+0000 7f89de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.676+0000 7f89de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.676+0000 7f89de7fc640 10 monclient: dump: 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:19.671 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:19.672 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:19.672 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:19.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.677+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f89d8003a20 con 0x7f89e80ae290 2024-03-27T03:06:19.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.678+0000 7f89edefc640 10 monclient: get_auth_request con 0x7f89cc06ddd0 auth_method 0 2024-03-27T03:06:19.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.678+0000 7f89f0160640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:19.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.679+0000 7f89f0160640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:19.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.679+0000 7f89f0160640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:19.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.679+0000 7f89de7fc640 10 client.4252.objecter ms_handle_connect 0x7f89cc06ddd0 2024-03-27T03:06:19.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.679+0000 7f89f0160640 1 -- 172.21.15.153:0/3052137472 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d0005080 con 0x7f89e80ae290 2024-03-27T03:06:19.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.679+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f89d8054410 con 0x7f89e80ae290 2024-03-27T03:06:19.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.679+0000 7f89de7fc640 10 client.4252.objecter ms_dispatch 0x7f89e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:06:19.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.679+0000 7f89de7fc640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:06:19.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.679+0000 7f89de7fc640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:06:19.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.679+0000 7f89de7fc640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-27T03:06:19.683 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.690+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89d80506b0 con 0x7f89e80ae290 2024-03-27T03:06:19.683 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.690+0000 7f89de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:19.683 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:19.690+0000 7f89de7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:06:20.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.025+0000 7f89f0160640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:20.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.025+0000 7f89f0160640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:20.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.025+0000 7f89f0160640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:20.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.025+0000 7f89f0160640 1 -- 172.21.15.153:0/3052137472 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89d0005330 con 0x7f89e80ae290 2024-03-27T03:06:20.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.026+0000 7f89de7fc640 1 -- 172.21.15.153:0/3052137472 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f89d80506b0 con 0x7f89e80ae290 2024-03-27T03:06:20.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.026+0000 7f89de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:20.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.026+0000 7f89de7fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:06:20.019 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:06:20.019 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:06:20.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.030+0000 7f89f0160640 10 librados: watch_flush enter 2024-03-27T03:06:20.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.031+0000 7f89f0160640 10 librados: watch_flush exit 2024-03-27T03:06:20.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.031+0000 7f89f0160640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-27T03:06:20.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.031+0000 7f89f0160640 10 client.4252.objecter successfully canceled tick 2024-03-27T03:06:20.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.031+0000 7f89f0160640 1 -- 172.21.15.153:0/3052137472 >> v1:172.21.15.153:6816/2637937781 conn(0x7f89cc06ddd0 legacy=0x7f89cc070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:20.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.031+0000 7f89f0160640 10 monclient: shutdown 2024-03-27T03:06:20.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.031+0000 7f89f0160640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:20.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.031+0000 7f89f0160640 1 -- 172.21.15.153:0/3052137472 >> v1:172.21.15.196:6789/0 conn(0x7f89e80ae290 legacy=0x7f89e8140d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:20.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.031+0000 7f89f0160640 1 -- 172.21.15.153:0/3052137472 shutdown_connections 2024-03-27T03:06:20.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.031+0000 7f89f0160640 1 -- 172.21.15.153:0/3052137472 >> 172.21.15.153:0/3052137472 conn(0x7f89e80b3c80 msgr2=0x7f89e80b1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:20.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.031+0000 7f89f0160640 1 -- 172.21.15.153:0/3052137472 shutdown_connections 2024-03-27T03:06:20.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.031+0000 7f89f0160640 1 -- 172.21.15.153:0/3052137472 wait complete. 2024-03-27T03:06:20.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:20.031+0000 7f89f0160640 1 librados: shutdown 2024-03-27T03:06:20.042 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:06:26.850 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:06:27.006 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.013+0000 7f604db4d640 10 monclient: get_monmap_and_config 2024-03-27T03:06:27.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.013+0000 7f604db4d640 10 monclient: build_initial_monmap 2024-03-27T03:06:27.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.014+0000 7f604db4d640 10 monclient: monmap: 2024-03-27T03:06:27.007 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:27.007 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:27.007 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:06:27.014921+0000 2024-03-27T03:06:27.007 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:06:27.014921+0000 2024-03-27T03:06:27.007 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:27.007 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:27.007 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:27.007 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:06:27.007 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:06:27.007 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:27.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.019+0000 7f604db4d640 1 Processor -- start 2024-03-27T03:06:27.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.020+0000 7f604db4d640 1 -- start start 2024-03-27T03:06:27.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.020+0000 7f604db4d640 10 monclient: init 2024-03-27T03:06:27.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.021+0000 7f604db4d640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:27.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.021+0000 7f604db4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:06:27.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.021+0000 7f604db4d640 10 monclient(hunting): picked mon.noname-c con 0x7f60481457f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:27.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.021+0000 7f604db4d640 10 monclient(hunting): picked mon.noname-b con 0x7f604813f650 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:27.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.021+0000 7f604db4d640 10 monclient(hunting): picked mon.noname-a con 0x7f60480a9d70 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:27.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.021+0000 7f604db4d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60480aaef0 con 0x7f60480a9d70 2024-03-27T03:06:27.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.021+0000 7f604db4d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6048145bd0 con 0x7f60481457f0 2024-03-27T03:06:27.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.021+0000 7f604db4d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6048145eb0 con 0x7f604813f650 2024-03-27T03:06:27.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.021+0000 7f604db4d640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:27.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.021+0000 7f604db4d640 10 monclient(hunting): authenticate will time out at 1347.327236s 2024-03-27T03:06:27.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.025+0000 7f60477fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f60481457f0 0x7f604813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:52839/0 (socket says 172.21.15.153:52839) 2024-03-27T03:06:27.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.025+0000 7f60477fe640 1 -- 172.21.15.153:0/1740419681 learned_addr learned my addr 172.21.15.153:0/1740419681 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:27.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.025+0000 7f60467fc640 1 -- 172.21.15.153:0/1740419681 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139226705 0 0) 0x7f6048145bd0 con 0x7f60481457f0 2024-03-27T03:06:27.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.025+0000 7f60467fc640 10 monclient(hunting): _negotiate 2024-03-27T03:06:27.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.025+0000 7f60467fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:27.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.026+0000 7f60467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:27.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.026+0000 7f60467fc640 10 monclient(hunting): my global_id is 4255 2024-03-27T03:06:27.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.026+0000 7f60467fc640 1 -- 172.21.15.153:0/1740419681 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6030003520 con 0x7f60481457f0 2024-03-27T03:06:27.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 1 -- 172.21.15.153:0/1740419681 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 608762914 0 0) 0x7f6030003520 con 0x7f60481457f0 2024-03-27T03:06:27.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 1 -- 172.21.15.153:0/1740419681 >> v1:172.21.15.153:6790/0 conn(0x7f604813f650 legacy=0x7f60480b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:06:27.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 1 -- 172.21.15.153:0/1740419681 >> v1:172.21.15.153:6789/0 conn(0x7f60480a9d70 legacy=0x7f604814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:27.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 10 monclient: _finish_hunting 0 2024-03-27T03:06:27.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 1 monclient: found mon.noname-c 2024-03-27T03:06:27.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:27.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:06:27.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 1 -- 172.21.15.153:0/1740419681 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6048146340 con 0x7f60481457f0 2024-03-27T03:06:27.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 10 monclient: _finish_auth 0 2024-03-27T03:06:27.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 10 monclient: _check_auth_tickets 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 1 -- 172.21.15.153:0/1740419681 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f603c001690 con 0x7f60481457f0 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.029+0000 7f60467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.030+0000 7f60467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.030+0000 7f60467fc640 10 monclient: dump: 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:27.025 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:27.026 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:27.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.030+0000 7f604db4d640 5 monclient: authenticate success, global_id 4255 2024-03-27T03:06:27.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.030+0000 7f604db4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:06:27.027 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.032+0000 7f60467fc640 1 -- 172.21.15.153:0/1740419681 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f603c001860 con 0x7f60481457f0 2024-03-27T03:06:27.027 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.032+0000 7f60467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:27.027 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.032+0000 7f60467fc640 1 -- 172.21.15.153:0/1740419681 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f603c001cb0 con 0x7f60481457f0 2024-03-27T03:06:27.027 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.032+0000 7f604db4d640 10 monclient: get_monmap_and_config success 2024-03-27T03:06:27.027 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.032+0000 7f604db4d640 4 set_mon_vals no callback set 2024-03-27T03:06:27.027 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.033+0000 7f604db4d640 10 monclient: shutdown 2024-03-27T03:06:27.027 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.034+0000 7f604db4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:27.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.034+0000 7f604db4d640 1 -- 172.21.15.153:0/1740419681 >> v1:172.21.15.196:6789/0 conn(0x7f60481457f0 legacy=0x7f604813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:27.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.034+0000 7f60467fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-27T03:06:27.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.035+0000 7f604db4d640 1 -- 172.21.15.153:0/1740419681 shutdown_connections 2024-03-27T03:06:27.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.035+0000 7f604db4d640 1 -- 172.21.15.153:0/1740419681 >> 172.21.15.153:0/1740419681 conn(0x7f6048066a00 msgr2=0x7f6048066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:27.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.035+0000 7f604db4d640 1 -- 172.21.15.153:0/1740419681 shutdown_connections 2024-03-27T03:06:27.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.036+0000 7f604db4d640 1 -- 172.21.15.153:0/1740419681 wait complete. 2024-03-27T03:06:27.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.037+0000 7f604db4d640 10 monclient: build_initial_monmap 2024-03-27T03:06:27.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.037+0000 7f604db4d640 10 monclient: monmap: 2024-03-27T03:06:27.032 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:27.032 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:06:27.033 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:06:27.033 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:06:27.033 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:27.033 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:27.033 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:27.033 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:06:27.033 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:06:27.033 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:27.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.037+0000 7f604db4d640 1 librados: starting msgr at 2024-03-27T03:06:27.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.037+0000 7f604db4d640 1 librados: starting objecter 2024-03-27T03:06:27.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.037+0000 7f604db4d640 1 Processor -- start 2024-03-27T03:06:27.034 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.038+0000 7f604db4d640 1 -- start start 2024-03-27T03:06:27.034 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.038+0000 7f604db4d640 1 librados: setting wanted keys 2024-03-27T03:06:27.034 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.038+0000 7f604db4d640 1 librados: calling monclient init 2024-03-27T03:06:27.034 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.038+0000 7f604db4d640 10 monclient: init 2024-03-27T03:06:27.034 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.039+0000 7f604db4d640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:27.034 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.039+0000 7f604db4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:06:27.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.040+0000 7f604db4d640 10 monclient(hunting): picked mon.noname-c con 0x7f60480a9d70 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:27.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.040+0000 7f604db4d640 10 monclient(hunting): picked mon.noname-a con 0x7f604813f650 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:27.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.040+0000 7f604db4d640 10 monclient(hunting): picked mon.noname-b con 0x7f60481457f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:27.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.040+0000 7f604db4d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6048140690 con 0x7f604813f650 2024-03-27T03:06:27.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.040+0000 7f604db4d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6048141910 con 0x7f60481457f0 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.040+0000 7f604db4d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6048145280 con 0x7f60480a9d70 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.040+0000 7f604db4d640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.040+0000 7f604db4d640 10 monclient(hunting): authenticate will time out at 1347.345667s 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.043+0000 7f60477fe640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f60480a9d70 0x7f6048134da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:56443/0 (socket says 172.21.15.153:56443) 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.043+0000 7f60477fe640 1 -- 172.21.15.153:0/607630545 learned_addr learned my addr 172.21.15.153:0/607630545 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.043+0000 7f602ffff640 10 client.?.objecter ms_handle_connect 0x7f60480a9d70 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.043+0000 7f602ffff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.043+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793846023 0 0) 0x7f6048145280 con 0x7f60480a9d70 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.044+0000 7f602ffff640 10 monclient(hunting): _negotiate 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.044+0000 7f602ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.044+0000 7f602ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.044+0000 7f602ffff640 10 monclient(hunting): my global_id is 4268 2024-03-27T03:06:27.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.044+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6034004eb0 con 0x7f60480a9d70 2024-03-27T03:06:27.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.045+0000 7f602ffff640 10 client.?.objecter ms_handle_connect 0x7f604813f650 2024-03-27T03:06:27.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.045+0000 7f602ffff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.045+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1793656399 0 0) 0x7f6034004eb0 con 0x7f60480a9d70 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.045+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6048145280 con 0x7f60480a9d70 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.045+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f603c001c30 con 0x7f60480a9d70 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 10 monclient(hunting): dump: 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:27.044 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 10 client.?.objecter ms_handle_connect 0x7f60481457f0 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3538200983 0 0) 0x7f6048145280 con 0x7f60480a9d70 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 >> v1:172.21.15.196:6789/0 conn(0x7f60481457f0 legacy=0x7f6048142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 >> v1:172.21.15.153:6789/0 conn(0x7f604813f650 legacy=0x7f6048142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 10 monclient: _finish_hunting 0 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 1 monclient: found mon.c 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60481354b0 con 0x7f60480a9d70 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.046+0000 7f602ffff640 10 monclient: _finish_auth 0 2024-03-27T03:06:27.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f602ffff640 10 monclient: _check_auth_tickets 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f602ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f603c004b00 con 0x7f60480a9d70 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f604db4d640 5 monclient: authenticate success, global_id 4268 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f604db4d640 10 monclient: _renew_subs 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f604db4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f604db4d640 1 -- 172.21.15.153:0/607630545 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6048135700 con 0x7f60480a9d70 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f604db4d640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f604db4d640 10 monclient: _renew_subs 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f604db4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f604db4d640 1 -- 172.21.15.153:0/607630545 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f604813df70 con 0x7f60480a9d70 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f604db4d640 1 librados: init done 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.047+0000 7f602ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.048+0000 7f6045ffb640 4 set_mon_vals no callback set 2024-03-27T03:06:27.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.048+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f603c004f10 con 0x7f60480a9d70 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.048+0000 7f602ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.048+0000 7f602ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.048+0000 7f602ffff640 10 monclient: dump: 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.049+0000 7f604db4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:27.047 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.049+0000 7f604db4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:27.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.049+0000 7f604db4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:06:27.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.049+0000 7f604db4d640 1 -- 172.21.15.153:0/607630545 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6028005080 con 0x7f60480a9d70 2024-03-27T03:06:27.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.050+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f603c0050f0 con 0x7f60480a9d70 2024-03-27T03:06:27.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.050+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f603c084c20 con 0x7f60480a9d70 2024-03-27T03:06:27.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.050+0000 7f602ffff640 10 client.4268.objecter ms_dispatch 0x7f6048000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:06:27.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.051+0000 7f602ffff640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:06:27.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.051+0000 7f602ffff640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:06:27.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.051+0000 7f602ffff640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-27T03:06:27.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.051+0000 7f6046ffd640 10 monclient: get_auth_request con 0x7f603406dd60 auth_method 0 2024-03-27T03:06:27.049 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.051+0000 7f602ffff640 10 client.4268.objecter ms_handle_connect 0x7f603406dd60 2024-03-27T03:06:27.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.059+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f603c050500 con 0x7f60480a9d70 2024-03-27T03:06:27.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.059+0000 7f602ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:27.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.059+0000 7f602ffff640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:06:27.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.388+0000 7f604db4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:27.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.388+0000 7f604db4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:27.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.388+0000 7f604db4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:06:27.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.388+0000 7f604db4d640 1 -- 172.21.15.153:0/607630545 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6028005330 con 0x7f60480a9d70 2024-03-27T03:06:27.383 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:06:27.383 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:06:27.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.390+0000 7f602ffff640 1 -- 172.21.15.153:0/607630545 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f603c052430 con 0x7f60480a9d70 2024-03-27T03:06:27.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.390+0000 7f602ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:27.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.390+0000 7f602ffff640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:06:27.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.393+0000 7f604db4d640 10 librados: watch_flush enter 2024-03-27T03:06:27.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.393+0000 7f604db4d640 10 librados: watch_flush exit 2024-03-27T03:06:27.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.393+0000 7f604db4d640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-27T03:06:27.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.393+0000 7f604db4d640 10 client.4268.objecter successfully canceled tick 2024-03-27T03:06:27.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.394+0000 7f604db4d640 1 -- 172.21.15.153:0/607630545 >> v1:172.21.15.153:6816/2637937781 conn(0x7f603406dd60 legacy=0x7f6034070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:27.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.394+0000 7f604db4d640 10 monclient: shutdown 2024-03-27T03:06:27.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.394+0000 7f604db4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:27.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.394+0000 7f604db4d640 1 -- 172.21.15.153:0/607630545 >> v1:172.21.15.153:6790/0 conn(0x7f60480a9d70 legacy=0x7f6048134da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:27.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.394+0000 7f604db4d640 1 -- 172.21.15.153:0/607630545 shutdown_connections 2024-03-27T03:06:27.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.394+0000 7f604db4d640 1 -- 172.21.15.153:0/607630545 >> 172.21.15.153:0/607630545 conn(0x7f6048066a00 msgr2=0x7f60480b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:27.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.395+0000 7f604db4d640 1 -- 172.21.15.153:0/607630545 shutdown_connections 2024-03-27T03:06:27.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.395+0000 7f604db4d640 1 -- 172.21.15.153:0/607630545 wait complete. 2024-03-27T03:06:27.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:27.395+0000 7f604db4d640 1 librados: shutdown 2024-03-27T03:06:27.409 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:06:34.216 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:06:34.367 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.374+0000 7f76f3e74640 10 monclient: get_monmap_and_config 2024-03-27T03:06:34.367 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.374+0000 7f76f3e74640 10 monclient: build_initial_monmap 2024-03-27T03:06:34.367 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.374+0000 7f76f3e74640 10 monclient: monmap: 2024-03-27T03:06:34.367 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:34.367 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:34.367 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:06:34.375265+0000 2024-03-27T03:06:34.367 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:06:34.375265+0000 2024-03-27T03:06:34.368 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:34.368 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:34.368 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:34.368 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:06:34.368 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:06:34.368 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:34.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.375+0000 7f76f3e74640 1 Processor -- start 2024-03-27T03:06:34.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.375+0000 7f76f3e74640 1 -- start start 2024-03-27T03:06:34.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.375+0000 7f76f3e74640 10 monclient: init 2024-03-27T03:06:34.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.376+0000 7f76f3e74640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:34.370 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.376+0000 7f76f3e74640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:06:34.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.376+0000 7f76f3e74640 10 monclient(hunting): picked mon.noname-b con 0x7f76ec1457f0 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:34.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.376+0000 7f76f3e74640 10 monclient(hunting): picked mon.noname-c con 0x7f76ec13f650 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:34.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.376+0000 7f76f3e74640 10 monclient(hunting): picked mon.noname-a con 0x7f76ec0a9d70 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:34.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.376+0000 7f76f3e74640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec0aaef0 con 0x7f76ec0a9d70 2024-03-27T03:06:34.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.376+0000 7f76f3e74640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec145bd0 con 0x7f76ec13f650 2024-03-27T03:06:34.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.376+0000 7f76f3e74640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec145eb0 con 0x7f76ec1457f0 2024-03-27T03:06:34.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.376+0000 7f76f3e74640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:34.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.376+0000 7f76f3e74640 10 monclient(hunting): authenticate will time out at 1354.682223s 2024-03-27T03:06:34.374 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.380+0000 7f76f1c10640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f76ec1457f0 0x7f76ec13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:34607/0 (socket says 172.21.15.153:34607) 2024-03-27T03:06:34.374 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.380+0000 7f76f1c10640 1 -- 172.21.15.153:0/847972546 learned_addr learned my addr 172.21.15.153:0/847972546 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:34.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 1 -- 172.21.15.153:0/847972546 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992798623 0 0) 0x7f76ec145eb0 con 0x7f76ec1457f0 2024-03-27T03:06:34.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 10 monclient(hunting): _negotiate 2024-03-27T03:06:34.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:34.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:34.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 10 monclient(hunting): my global_id is 4271 2024-03-27T03:06:34.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 1 -- 172.21.15.153:0/847972546 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d8003520 con 0x7f76ec1457f0 2024-03-27T03:06:34.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 1 -- 172.21.15.153:0/847972546 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2729429902 0 0) 0x7f76d8003520 con 0x7f76ec1457f0 2024-03-27T03:06:34.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 1 -- 172.21.15.153:0/847972546 >> v1:172.21.15.196:6789/0 conn(0x7f76ec13f650 legacy=0x7f76ec0b2620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 1 -- 172.21.15.153:0/847972546 >> v1:172.21.15.153:6789/0 conn(0x7f76ec0a9d70 legacy=0x7f76ec14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 10 monclient: _finish_hunting 0 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 1 monclient: found mon.noname-b 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.382+0000 7f76f0c0e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 1 -- 172.21.15.153:0/847972546 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec146340 con 0x7f76ec1457f0 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 10 monclient: _finish_auth 0 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 10 monclient: _check_auth_tickets 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 1 -- 172.21.15.153:0/847972546 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f76e4001a10 con 0x7f76ec1457f0 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:06:34.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 10 monclient: dump: 2024-03-27T03:06:34.380 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:34.380 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:34.380 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:34.380 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:34.380 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:34.380 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:34.380 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:34.380 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:34.380 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:34.380 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:34.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 1 -- 172.21.15.153:0/847972546 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76e4001ca0 con 0x7f76ec1457f0 2024-03-27T03:06:34.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:34.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 1 -- 172.21.15.153:0/847972546 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f76e4002030 con 0x7f76ec1457f0 2024-03-27T03:06:34.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:34.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:06:34.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.383+0000 7f76f0c0e640 10 monclient: dump: 2024-03-27T03:06:34.381 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:34.381 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:34.381 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:34.381 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:34.381 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:34.381 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:34.381 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:34.381 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:34.382 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:34.382 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:34.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.384+0000 7f76f3e74640 5 monclient: authenticate success, global_id 4271 2024-03-27T03:06:34.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.384+0000 7f76f3e74640 10 monclient: get_monmap_and_config success 2024-03-27T03:06:34.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.384+0000 7f76f3e74640 4 set_mon_vals no callback set 2024-03-27T03:06:34.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.384+0000 7f76f3e74640 10 monclient: shutdown 2024-03-27T03:06:34.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.384+0000 7f76f3e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:34.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.384+0000 7f76f3e74640 1 -- 172.21.15.153:0/847972546 >> v1:172.21.15.153:6790/0 conn(0x7f76ec1457f0 legacy=0x7f76ec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:34.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.385+0000 7f76f3e74640 1 -- 172.21.15.153:0/847972546 shutdown_connections 2024-03-27T03:06:34.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.385+0000 7f76f3e74640 1 -- 172.21.15.153:0/847972546 >> 172.21.15.153:0/847972546 conn(0x7f76ec066a00 msgr2=0x7f76ec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:34.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.389+0000 7f76f3e74640 1 -- 172.21.15.153:0/847972546 shutdown_connections 2024-03-27T03:06:34.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.389+0000 7f76f3e74640 1 -- 172.21.15.153:0/847972546 wait complete. 2024-03-27T03:06:34.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.390+0000 7f76f3e74640 10 monclient: build_initial_monmap 2024-03-27T03:06:34.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.390+0000 7f76f3e74640 10 monclient: monmap: 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.391+0000 7f76f3e74640 1 librados: starting msgr at 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.391+0000 7f76f3e74640 1 librados: starting objecter 2024-03-27T03:06:34.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.391+0000 7f76f3e74640 1 Processor -- start 2024-03-27T03:06:34.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.391+0000 7f76f3e74640 1 -- start start 2024-03-27T03:06:34.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.391+0000 7f76f3e74640 1 librados: setting wanted keys 2024-03-27T03:06:34.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.391+0000 7f76f3e74640 1 librados: calling monclient init 2024-03-27T03:06:34.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.391+0000 7f76f3e74640 10 monclient: init 2024-03-27T03:06:34.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.392+0000 7f76f3e74640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:34.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.392+0000 7f76f3e74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:06:34.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.396+0000 7f76f3e74640 10 monclient(hunting): picked mon.noname-b con 0x7f76ec0a9d70 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:34.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.396+0000 7f76f1c10640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f76ec0a9d70 0x7f76ec1398d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:58827/0 (socket says 172.21.15.153:58827) 2024-03-27T03:06:34.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.396+0000 7f76f1c10640 1 -- 172.21.15.153:0/3741507126 learned_addr learned my addr 172.21.15.153:0/3741507126 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:34.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.396+0000 7f76f3e74640 10 monclient(hunting): picked mon.noname-c con 0x7f76ec13f650 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:34.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.399+0000 7f76f3e74640 10 monclient(hunting): picked mon.noname-a con 0x7f76ec1457f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:34.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.399+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec13a230 con 0x7f76ec1457f0 2024-03-27T03:06:34.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.399+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec13b2e0 con 0x7f76ec0a9d70 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.399+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec138e60 con 0x7f76ec13f650 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.399+0000 7f76f3e74640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.399+0000 7f76f3e74640 10 monclient(hunting): authenticate will time out at 1354.705243s 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.399+0000 7f76e27fc640 10 client.?.objecter ms_handle_connect 0x7f76ec0a9d70 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.399+0000 7f76e27fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.400+0000 7f76e27fc640 10 client.?.objecter ms_handle_connect 0x7f76ec13f650 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.400+0000 7f76e27fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.400+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272875946 0 0) 0x7f76ec138e60 con 0x7f76ec13f650 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.400+0000 7f76e27fc640 10 monclient(hunting): _negotiate 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.400+0000 7f76e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.400+0000 7f76e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.400+0000 7f76e27fc640 10 monclient(hunting): my global_id is 4274 2024-03-27T03:06:34.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.400+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d0004eb0 con 0x7f76ec13f650 2024-03-27T03:06:34.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.400+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228386506 0 0) 0x7f76ec13b2e0 con 0x7f76ec0a9d70 2024-03-27T03:06:34.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.400+0000 7f76e27fc640 10 monclient(hunting): _negotiate 2024-03-27T03:06:34.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.400+0000 7f76e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:34.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.401+0000 7f76e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:34.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.401+0000 7f76e27fc640 10 monclient(hunting): my global_id is 4261 2024-03-27T03:06:34.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.401+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76ec138e60 con 0x7f76ec0a9d70 2024-03-27T03:06:34.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.401+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4061627748 0 0) 0x7f76d0004eb0 con 0x7f76ec13f650 2024-03-27T03:06:34.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.401+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76ec13b2e0 con 0x7f76ec13f650 2024-03-27T03:06:34.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.401+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f76dc001870 con 0x7f76ec13f650 2024-03-27T03:06:34.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.401+0000 7f76e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:34.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.401+0000 7f76e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:06:34.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.401+0000 7f76e27fc640 10 monclient(hunting): dump: 2024-03-27T03:06:34.399 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:34.399 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:34.399 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:34.399 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:34.399 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:34.399 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:34.399 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:34.399 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:34.399 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:34.399 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:34.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.401+0000 7f76e27fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:06:34.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.401+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1843383013 0 0) 0x7f76ec13b2e0 con 0x7f76ec13f650 2024-03-27T03:06:34.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.401+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 >> v1:172.21.15.196:6789/0 conn(0x7f76ec0a9d70 legacy=0x7f76ec1398d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:34.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 >> v1:172.21.15.153:6789/0 conn(0x7f76ec1457f0 legacy=0x7f76ec1d7b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:06:34.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 10 monclient: _finish_hunting 0 2024-03-27T03:06:34.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 1 monclient: found mon.c 2024-03-27T03:06:34.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:34.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:06:34.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec13b9c0 con 0x7f76ec13f650 2024-03-27T03:06:34.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 10 monclient: _finish_auth 0 2024-03-27T03:06:34.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 10 monclient: _check_auth_tickets 2024-03-27T03:06:34.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:34.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76f3e74640 5 monclient: authenticate success, global_id 4274 2024-03-27T03:06:34.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76f3e74640 10 monclient: _renew_subs 2024-03-27T03:06:34.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76f3e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:06:34.402 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ec13bbb0 con 0x7f76ec13f650 2024-03-27T03:06:34.402 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76f3e74640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:06:34.402 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76f3e74640 10 monclient: _renew_subs 2024-03-27T03:06:34.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76f3e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:06:34.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ec1e2f50 con 0x7f76ec13f650 2024-03-27T03:06:34.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76f3e74640 1 librados: init done 2024-03-27T03:06:34.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76dc001d80 con 0x7f76ec13f650 2024-03-27T03:06:34.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:34.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f76dc003830 con 0x7f76ec13f650 2024-03-27T03:06:34.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:34.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:06:34.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.402+0000 7f76e27fc640 10 monclient: dump: 2024-03-27T03:06:34.403 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:34.403 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.403+0000 7f76e3fff640 4 set_mon_vals no callback set 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.405+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f76dc003990 con 0x7f76ec13f650 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.405+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f76dc083800 con 0x7f76ec13f650 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.405+0000 7f76e27fc640 10 client.4274.objecter ms_dispatch 0x7f76ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:06:34.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.405+0000 7f76e27fc640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:06:34.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.405+0000 7f76e27fc640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:06:34.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.405+0000 7f76e27fc640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-27T03:06:34.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.406+0000 7f76f3e74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:34.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.406+0000 7f76f3e74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:34.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.406+0000 7f76f3e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:06:34.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.406+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d4005080 con 0x7f76ec13f650 2024-03-27T03:06:34.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.406+0000 7f76f1c10640 10 monclient: get_auth_request con 0x7f76d006da30 auth_method 0 2024-03-27T03:06:34.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.407+0000 7f76e27fc640 10 client.4274.objecter ms_handle_connect 0x7f76d006da30 2024-03-27T03:06:34.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.415+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76dc04ef10 con 0x7f76ec13f650 2024-03-27T03:06:34.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.415+0000 7f76e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:34.408 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.415+0000 7f76e27fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:06:34.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.739+0000 7f76f3e74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:34.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.739+0000 7f76f3e74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:34.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.739+0000 7f76f3e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:06:34.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.739+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76d4005330 con 0x7f76ec13f650 2024-03-27T03:06:34.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.740+0000 7f76e27fc640 1 -- 172.21.15.153:0/3741507126 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f76dc051010 con 0x7f76ec13f650 2024-03-27T03:06:34.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.740+0000 7f76e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:34.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.740+0000 7f76e27fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:06:34.733 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:06:34.734 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:06:34.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.743+0000 7f76f3e74640 10 librados: watch_flush enter 2024-03-27T03:06:34.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.744+0000 7f76f3e74640 10 librados: watch_flush exit 2024-03-27T03:06:34.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.744+0000 7f76f3e74640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-27T03:06:34.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.744+0000 7f76f3e74640 10 client.4274.objecter successfully canceled tick 2024-03-27T03:06:34.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.746+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 >> v1:172.21.15.153:6816/2637937781 conn(0x7f76d006da30 legacy=0x7f76d006fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:34.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.746+0000 7f76f3e74640 10 monclient: shutdown 2024-03-27T03:06:34.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.746+0000 7f76f3e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:34.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.746+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 >> v1:172.21.15.153:6790/0 conn(0x7f76ec13f650 legacy=0x7f76ec1d7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:34.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.746+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 shutdown_connections 2024-03-27T03:06:34.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.746+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 >> 172.21.15.153:0/3741507126 conn(0x7f76ec066a00 msgr2=0x7f76ec0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:34.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.746+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 shutdown_connections 2024-03-27T03:06:34.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.746+0000 7f76f3e74640 1 -- 172.21.15.153:0/3741507126 wait complete. 2024-03-27T03:06:34.740 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:34.746+0000 7f76f3e74640 1 librados: shutdown 2024-03-27T03:06:34.758 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:06:41.565 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:06:41.748 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.755+0000 7f6289fed640 10 monclient: get_monmap_and_config 2024-03-27T03:06:41.748 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.755+0000 7f6289fed640 10 monclient: build_initial_monmap 2024-03-27T03:06:41.748 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.755+0000 7f6289fed640 10 monclient: monmap: 2024-03-27T03:06:41.748 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:41.748 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:41.749 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:06:41.756348+0000 2024-03-27T03:06:41.749 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:06:41.756348+0000 2024-03-27T03:06:41.749 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:41.749 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:41.749 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:41.749 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:06:41.749 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:06:41.749 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:41.749 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.756+0000 7f6289fed640 1 Processor -- start 2024-03-27T03:06:41.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.756+0000 7f6289fed640 1 -- start start 2024-03-27T03:06:41.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.756+0000 7f6289fed640 10 monclient: init 2024-03-27T03:06:41.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.757+0000 7f6289fed640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:41.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.757+0000 7f6289fed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:06:41.752 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.757+0000 7f6289fed640 10 monclient(hunting): picked mon.noname-a con 0x7f62841457f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:41.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.757+0000 7f6289fed640 10 monclient(hunting): picked mon.noname-b con 0x7f628413f650 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:41.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.757+0000 7f6289fed640 10 monclient(hunting): picked mon.noname-c con 0x7f62840a9d70 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:41.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.757+0000 7f6289fed640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62840aaef0 con 0x7f62841457f0 2024-03-27T03:06:41.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.757+0000 7f6289fed640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6284145bd0 con 0x7f62840a9d70 2024-03-27T03:06:41.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.757+0000 7f6289fed640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6284145eb0 con 0x7f628413f650 2024-03-27T03:06:41.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.757+0000 7f6289fed640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:41.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.757+0000 7f6289fed640 10 monclient(hunting): authenticate will time out at 1362.063535s 2024-03-27T03:06:41.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.758+0000 7f62837fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f62841457f0 0x7f628413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:43768/0 (socket says 172.21.15.153:43768) 2024-03-27T03:06:41.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.758+0000 7f62837fe640 1 -- 172.21.15.153:0/576187050 learned_addr learned my addr 172.21.15.153:0/576187050 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:41.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808139751 0 0) 0x7f62840aaef0 con 0x7f62841457f0 2024-03-27T03:06:41.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient(hunting): _negotiate 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient(hunting): my global_id is 4254 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f626c003520 con 0x7f62841457f0 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287504642 0 0) 0x7f6284145bd0 con 0x7f62840a9d70 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient(hunting): _negotiate 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient(hunting): my global_id is 4264 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62840aaef0 con 0x7f62840a9d70 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189276941 0 0) 0x7f6284145eb0 con 0x7f628413f650 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient(hunting): _negotiate 2024-03-27T03:06:41.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient(hunting): my global_id is 4277 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6284145bd0 con 0x7f628413f650 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3358667980 0 0) 0x7f62840aaef0 con 0x7f62840a9d70 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 >> v1:172.21.15.153:6790/0 conn(0x7f628413f650 legacy=0x7f62840b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 >> v1:172.21.15.153:6789/0 conn(0x7f62841457f0 legacy=0x7f628413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient: _finish_hunting 0 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 1 monclient: found mon.noname-c 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6284146340 con 0x7f62840a9d70 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient: _finish_auth 0 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient: _check_auth_tickets 2024-03-27T03:06:41.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f62740015f0 con 0x7f62840a9d70 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.759+0000 7f62827fc640 10 monclient: dump: 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.760+0000 7f6289fed640 5 monclient: authenticate success, global_id 4264 2024-03-27T03:06:41.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.760+0000 7f6289fed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:06:41.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.760+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6274001750 con 0x7f62840a9d70 2024-03-27T03:06:41.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.760+0000 7f62827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:41.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.760+0000 7f6289fed640 10 monclient: get_monmap_and_config success 2024-03-27T03:06:41.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.760+0000 7f6289fed640 4 set_mon_vals no callback set 2024-03-27T03:06:41.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.760+0000 7f62827fc640 1 -- 172.21.15.153:0/576187050 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6274001b60 con 0x7f62840a9d70 2024-03-27T03:06:41.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.760+0000 7f62827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:41.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.760+0000 7f62827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:06:41.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.760+0000 7f62827fc640 10 monclient: dump: 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.761+0000 7f6289fed640 10 monclient: shutdown 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.761+0000 7f6289fed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.761+0000 7f6289fed640 1 -- 172.21.15.153:0/576187050 >> v1:172.21.15.196:6789/0 conn(0x7f62840a9d70 legacy=0x7f628414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.761+0000 7f6289fed640 1 -- 172.21.15.153:0/576187050 shutdown_connections 2024-03-27T03:06:41.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.761+0000 7f6289fed640 1 -- 172.21.15.153:0/576187050 >> 172.21.15.153:0/576187050 conn(0x7f6284066a00 msgr2=0x7f6284066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.762+0000 7f6289fed640 1 -- 172.21.15.153:0/576187050 shutdown_connections 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.762+0000 7f6289fed640 1 -- 172.21.15.153:0/576187050 wait complete. 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.762+0000 7f6289fed640 10 monclient: build_initial_monmap 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.762+0000 7f6289fed640 10 monclient: monmap: 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:41.762 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.763+0000 7f6289fed640 1 librados: starting msgr at 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.763+0000 7f6289fed640 1 librados: starting objecter 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.763+0000 7f6289fed640 1 Processor -- start 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.763+0000 7f6289fed640 1 -- start start 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.763+0000 7f6289fed640 1 librados: setting wanted keys 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.763+0000 7f6289fed640 1 librados: calling monclient init 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.763+0000 7f6289fed640 10 monclient: init 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f6289fed640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f6289fed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f6289fed640 10 monclient(hunting): picked mon.noname-a con 0x7f62840a9d70 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f6289fed640 10 monclient(hunting): picked mon.noname-c con 0x7f628413f650 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f6289fed640 10 monclient(hunting): picked mon.noname-b con 0x7f62841457f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f6289fed640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6284139940 con 0x7f62840a9d70 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f6289fed640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62841ce970 con 0x7f62841457f0 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f6283fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f62841457f0 0x7f6284139230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:34314/0 (socket says 172.21.15.153:34314) 2024-03-27T03:06:41.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f6283fff640 1 -- 172.21.15.153:0/2776781739 learned_addr learned my addr 172.21.15.153:0/2776781739 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f6289fed640 1 -- 172.21.15.153:0/2776781739 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62841cec50 con 0x7f628413f650 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f6289fed640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f626bfff640 10 client.?.objecter ms_handle_connect 0x7f62841457f0 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f626bfff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.764+0000 7f6289fed640 10 monclient(hunting): authenticate will time out at 1362.070537s 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.765+0000 7f626bfff640 10 client.?.objecter ms_handle_connect 0x7f62840a9d70 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.765+0000 7f626bfff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.765+0000 7f626bfff640 10 client.?.objecter ms_handle_connect 0x7f628413f650 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.765+0000 7f626bfff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.765+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872919437 0 0) 0x7f62841ce970 con 0x7f62841457f0 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.765+0000 7f626bfff640 10 monclient(hunting): _negotiate 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.765+0000 7f626bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.765+0000 7f626bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.765+0000 7f626bfff640 10 monclient(hunting): my global_id is 4267 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.765+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6270004eb0 con 0x7f62841457f0 2024-03-27T03:06:41.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080633815 0 0) 0x7f62841cec50 con 0x7f628413f650 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): _negotiate 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): my global_id is 4280 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62841ce970 con 0x7f628413f650 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635686697 0 0) 0x7f6284139940 con 0x7f62840a9d70 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): _negotiate 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): my global_id is 4257 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6270005360 con 0x7f62840a9d70 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3427140309 0 0) 0x7f62841ce970 con 0x7f628413f650 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6284139940 con 0x7f628413f650 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6278001860 con 0x7f628413f650 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:41.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): dump: 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:06:41.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2922141877 0 0) 0x7f6270004eb0 con 0x7f62841457f0 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62841ce970 con 0x7f62841457f0 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6274001aa0 con 0x7f62841457f0 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.766+0000 7f626bfff640 10 monclient(hunting): dump: 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:41.767 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1317653718 0 0) 0x7f62841ce970 con 0x7f62841457f0 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 >> v1:172.21.15.153:6790/0 conn(0x7f628413f650 legacy=0x7f6284138b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 >> v1:172.21.15.153:6789/0 conn(0x7f62840a9d70 legacy=0x7f62841cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 10 monclient: _finish_hunting 0 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 1 monclient: found mon.b 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62841cef30 con 0x7f62841457f0 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 10 monclient: _finish_auth 0 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 10 monclient: _check_auth_tickets 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f6289fed640 5 monclient: authenticate success, global_id 4267 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f6289fed640 10 monclient: _renew_subs 2024-03-27T03:06:41.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f6289fed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f6289fed640 1 -- 172.21.15.153:0/2776781739 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62841cc990 con 0x7f62841457f0 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f6289fed640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f6289fed640 10 monclient: _renew_subs 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f6289fed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f6289fed640 1 -- 172.21.15.153:0/2776781739 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62841cd910 con 0x7f62841457f0 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f6289fed640 1 librados: init done 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62740038a0 con 0x7f62841457f0 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6274003c30 con 0x7f62841457f0 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f626bfff640 10 monclient: dump: 2024-03-27T03:06:41.769 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:41.770 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:41.770 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:41.770 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:41.770 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:41.770 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:41.770 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:41.770 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:41.770 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:41.770 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:41.770 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.767+0000 7f6281ffb640 4 set_mon_vals no callback set 2024-03-27T03:06:41.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.769+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f6274003d90 con 0x7f62841457f0 2024-03-27T03:06:41.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.770+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f6274084b10 con 0x7f62841457f0 2024-03-27T03:06:41.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.770+0000 7f626bfff640 10 client.4267.objecter ms_dispatch 0x7f6284000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:06:41.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.770+0000 7f626bfff640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:06:41.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.770+0000 7f626bfff640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:06:41.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.770+0000 7f626bfff640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-27T03:06:41.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.770+0000 7f62837fe640 10 monclient: get_auth_request con 0x7f627006e1a0 auth_method 0 2024-03-27T03:06:41.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.770+0000 7f626bfff640 10 client.4267.objecter ms_handle_connect 0x7f627006e1a0 2024-03-27T03:06:41.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.771+0000 7f6289fed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:41.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.771+0000 7f6289fed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:41.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.771+0000 7f6289fed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:41.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.771+0000 7f6289fed640 1 -- 172.21.15.153:0/2776781739 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6264005080 con 0x7f62841457f0 2024-03-27T03:06:41.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.779+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62740503f0 con 0x7f62841457f0 2024-03-27T03:06:41.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.779+0000 7f626bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:41.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:41.779+0000 7f626bfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:06:42.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.160+0000 7f6289fed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:42.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.160+0000 7f6289fed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:42.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.160+0000 7f6289fed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:42.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.160+0000 7f6289fed640 1 -- 172.21.15.153:0/2776781739 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6264005330 con 0x7f62841457f0 2024-03-27T03:06:42.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.161+0000 7f626bfff640 1 -- 172.21.15.153:0/2776781739 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f6274052320 con 0x7f62841457f0 2024-03-27T03:06:42.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.161+0000 7f626bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:42.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.161+0000 7f626bfff640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:06:42.154 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:06:42.154 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:06:42.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.164+0000 7f6289fed640 10 librados: watch_flush enter 2024-03-27T03:06:42.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.164+0000 7f6289fed640 10 librados: watch_flush exit 2024-03-27T03:06:42.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.164+0000 7f6289fed640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-27T03:06:42.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.164+0000 7f6289fed640 10 client.4267.objecter successfully canceled tick 2024-03-27T03:06:42.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.164+0000 7f6289fed640 1 -- 172.21.15.153:0/2776781739 >> v1:172.21.15.153:6816/2637937781 conn(0x7f627006e1a0 legacy=0x7f6270070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:42.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.164+0000 7f6289fed640 10 monclient: shutdown 2024-03-27T03:06:42.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.164+0000 7f6289fed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:42.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.164+0000 7f6289fed640 1 -- 172.21.15.153:0/2776781739 >> v1:172.21.15.196:6789/0 conn(0x7f62841457f0 legacy=0x7f6284139230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:42.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.165+0000 7f6289fed640 1 -- 172.21.15.153:0/2776781739 shutdown_connections 2024-03-27T03:06:42.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.165+0000 7f6289fed640 1 -- 172.21.15.153:0/2776781739 >> 172.21.15.153:0/2776781739 conn(0x7f6284066a00 msgr2=0x7f6284066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:42.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.165+0000 7f6289fed640 1 -- 172.21.15.153:0/2776781739 shutdown_connections 2024-03-27T03:06:42.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.165+0000 7f6289fed640 1 -- 172.21.15.153:0/2776781739 wait complete. 2024-03-27T03:06:42.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:42.165+0000 7f6289fed640 1 librados: shutdown 2024-03-27T03:06:42.173 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:06:48.980 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:06:49.127 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.134+0000 7ffaa3910640 10 monclient: get_monmap_and_config 2024-03-27T03:06:49.127 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.134+0000 7ffaa3910640 10 monclient: build_initial_monmap 2024-03-27T03:06:49.127 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.134+0000 7ffaa3910640 10 monclient: monmap: 2024-03-27T03:06:49.128 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:49.128 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:49.128 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:06:49.135539+0000 2024-03-27T03:06:49.128 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:06:49.135539+0000 2024-03-27T03:06:49.128 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:49.128 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:49.128 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:49.128 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:06:49.128 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:06:49.128 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:49.129 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.135+0000 7ffaa3910640 1 Processor -- start 2024-03-27T03:06:49.129 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.136+0000 7ffaa3910640 1 -- start start 2024-03-27T03:06:49.129 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.136+0000 7ffaa3910640 10 monclient: init 2024-03-27T03:06:49.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.137+0000 7ffaa3910640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:49.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.137+0000 7ffaa3910640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:06:49.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.137+0000 7ffaa3910640 10 monclient(hunting): picked mon.noname-a con 0x7ffa9c0b0770 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:49.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.137+0000 7ffaa3910640 10 monclient(hunting): picked mon.noname-b con 0x7ffa9c14a700 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:49.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.137+0000 7ffaa3910640 10 monclient(hunting): picked mon.noname-c con 0x7ffa9c0ac5a0 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:49.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.137+0000 7ffaa3910640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c0aba70 con 0x7ffa9c0b0770 2024-03-27T03:06:49.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.137+0000 7ffaa3910640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c0aecb0 con 0x7ffa9c0ac5a0 2024-03-27T03:06:49.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.137+0000 7ffaa3910640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c0af150 con 0x7ffa9c14a700 2024-03-27T03:06:49.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.137+0000 7ffaa3910640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:49.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.137+0000 7ffaa3910640 10 monclient(hunting): authenticate will time out at 1369.443000s 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.137+0000 7ffaa1ead640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ffa9c0ac5a0 0x7ffa9c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:49914/0 (socket says 172.21.15.153:49914) 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.137+0000 7ffaa1ead640 1 -- 172.21.15.153:0/1752768559 learned_addr learned my addr 172.21.15.153:0/1752768559 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.138+0000 7ffa9bfff640 1 -- 172.21.15.153:0/1752768559 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737525682 0 0) 0x7ffa9c0aba70 con 0x7ffa9c0b0770 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.138+0000 7ffa9bfff640 10 monclient(hunting): _negotiate 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.138+0000 7ffa9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.138+0000 7ffa9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.138+0000 7ffa9bfff640 10 monclient(hunting): my global_id is 4260 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.138+0000 7ffa9bfff640 1 -- 172.21.15.153:0/1752768559 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa84003520 con 0x7ffa9c0b0770 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.138+0000 7ffa9bfff640 1 -- 172.21.15.153:0/1752768559 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694176473 0 0) 0x7ffa9c0aecb0 con 0x7ffa9c0ac5a0 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.138+0000 7ffa9bfff640 10 monclient(hunting): _negotiate 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.138+0000 7ffa9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.138+0000 7ffa9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.138+0000 7ffa9bfff640 10 monclient(hunting): my global_id is 4270 2024-03-27T03:06:49.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.138+0000 7ffa9bfff640 1 -- 172.21.15.153:0/1752768559 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9c0aba70 con 0x7ffa9c0ac5a0 2024-03-27T03:06:49.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 1 -- 172.21.15.153:0/1752768559 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3906015712 0 0) 0x7ffa84003520 con 0x7ffa9c0b0770 2024-03-27T03:06:49.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 1 -- 172.21.15.153:0/1752768559 >> v1:172.21.15.153:6790/0 conn(0x7ffa9c14a700 legacy=0x7ffa9c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:49.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 1 -- 172.21.15.153:0/1752768559 >> v1:172.21.15.196:6789/0 conn(0x7ffa9c0ac5a0 legacy=0x7ffa9c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:49.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 10 monclient: _finish_hunting 0 2024-03-27T03:06:49.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 1 monclient: found mon.noname-a 2024-03-27T03:06:49.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:49.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:06:49.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 1 -- 172.21.15.153:0/1752768559 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c0ac9d0 con 0x7ffa9c0b0770 2024-03-27T03:06:49.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 10 monclient: _finish_auth 0 2024-03-27T03:06:49.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 10 monclient: _check_auth_tickets 2024-03-27T03:06:49.135 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:49.135 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 1 -- 172.21.15.153:0/1752768559 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ffa94001690 con 0x7ffa9c0b0770 2024-03-27T03:06:49.135 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:49.135 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:06:49.135 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 10 monclient: dump: 2024-03-27T03:06:49.135 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:49.135 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:49.135 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:49.135 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:49.135 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:49.135 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffaa3910640 5 monclient: authenticate success, global_id 4260 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffaa3910640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 1 -- 172.21.15.153:0/1752768559 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa940017f0 con 0x7ffa9c0b0770 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 1 -- 172.21.15.153:0/1752768559 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ffa94001c40 con 0x7ffa9c0b0770 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.139+0000 7ffa9bfff640 10 monclient: dump: 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:49.136 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:49.137 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:49.137 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:49.137 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:49.137 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:49.137 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:49.137 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:49.137 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:49.137 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.140+0000 7ffaa3910640 10 monclient: get_monmap_and_config success 2024-03-27T03:06:49.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.140+0000 7ffaa3910640 4 set_mon_vals no callback set 2024-03-27T03:06:49.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.140+0000 7ffaa3910640 10 monclient: shutdown 2024-03-27T03:06:49.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.140+0000 7ffaa3910640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:49.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.140+0000 7ffaa3910640 1 -- 172.21.15.153:0/1752768559 >> v1:172.21.15.153:6789/0 conn(0x7ffa9c0b0770 legacy=0x7ffa9c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:49.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.140+0000 7ffaa3910640 1 -- 172.21.15.153:0/1752768559 shutdown_connections 2024-03-27T03:06:49.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.141+0000 7ffaa3910640 1 -- 172.21.15.153:0/1752768559 >> 172.21.15.153:0/1752768559 conn(0x7ffa9c06b660 msgr2=0x7ffa9c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:49.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.141+0000 7ffaa3910640 1 -- 172.21.15.153:0/1752768559 shutdown_connections 2024-03-27T03:06:49.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.141+0000 7ffaa3910640 1 -- 172.21.15.153:0/1752768559 wait complete. 2024-03-27T03:06:49.146 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.142+0000 7ffaa3910640 10 monclient: build_initial_monmap 2024-03-27T03:06:49.146 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.142+0000 7ffaa3910640 10 monclient: monmap: 2024-03-27T03:06:49.146 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:49.146 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:06:49.146 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:06:49.146 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:06:49.146 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:49.146 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:49.146 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:49.146 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:06:49.146 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:06:49.146 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.142+0000 7ffaa3910640 1 librados: starting msgr at 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.142+0000 7ffaa3910640 1 librados: starting objecter 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.142+0000 7ffaa3910640 1 Processor -- start 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.142+0000 7ffaa3910640 1 -- start start 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.142+0000 7ffaa3910640 1 librados: setting wanted keys 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.142+0000 7ffaa3910640 1 librados: calling monclient init 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.142+0000 7ffaa3910640 10 monclient: init 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.143+0000 7ffaa3910640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.143+0000 7ffaa3910640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.143+0000 7ffaa3910640 10 monclient(hunting): picked mon.noname-c con 0x7ffa9c0ac5a0 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.143+0000 7ffaa3910640 10 monclient(hunting): picked mon.noname-a con 0x7ffa9c0b0770 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.143+0000 7ffaa3910640 10 monclient(hunting): picked mon.noname-b con 0x7ffa9c14a700 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.143+0000 7ffaa3910640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c1dfab0 con 0x7ffa9c0b0770 2024-03-27T03:06:49.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffaa1ead640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ffa9c14a700 0x7ffa9c1e64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:50672/0 (socket says 172.21.15.153:50672) 2024-03-27T03:06:49.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffaa1ead640 1 -- 172.21.15.153:0/2101296648 learned_addr learned my addr 172.21.15.153:0/2101296648 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:49.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffaa3910640 1 -- 172.21.15.153:0/2101296648 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c128940 con 0x7ffa9c14a700 2024-03-27T03:06:49.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffaa3910640 1 -- 172.21.15.153:0/2101296648 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c145b10 con 0x7ffa9c0ac5a0 2024-03-27T03:06:49.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 client.?.objecter ms_handle_connect 0x7ffa9c14a700 2024-03-27T03:06:49.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:49.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 client.?.objecter ms_handle_connect 0x7ffa9c0ac5a0 2024-03-27T03:06:49.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:49.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffaa3910640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:49.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffaa3910640 10 monclient(hunting): authenticate will time out at 1369.450143s 2024-03-27T03:06:49.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932863681 0 0) 0x7ffa9c128940 con 0x7ffa9c14a700 2024-03-27T03:06:49.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:06:49.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 monclient(hunting): my global_id is 4273 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa88004eb0 con 0x7ffa9c14a700 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 client.?.objecter ms_handle_connect 0x7ffa9c0b0770 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803239098 0 0) 0x7ffa9c145b10 con 0x7ffa9c0ac5a0 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 10 monclient(hunting): my global_id is 4286 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.144+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9c128940 con 0x7ffa9c0ac5a0 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4171520474 0 0) 0x7ffa88004eb0 con 0x7ffa9c14a700 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa9c145b10 con 0x7ffa9c14a700 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ffa8c001d90 con 0x7ffa9c14a700 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:49.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient(hunting): dump: 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566805862 0 0) 0x7ffa9c1dfab0 con 0x7ffa9c0b0770 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:06:49.150 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient(hunting): my global_id is 4263 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa88004eb0 con 0x7ffa9c0b0770 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 425211924 0 0) 0x7ffa9c128940 con 0x7ffa9c0ac5a0 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa9c1dfab0 con 0x7ffa9c0ac5a0 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ffa94001b00 con 0x7ffa9c0ac5a0 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient(hunting): dump: 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:49.151 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2628272686 0 0) 0x7ffa9c145b10 con 0x7ffa9c14a700 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 >> v1:172.21.15.153:6790/0 conn(0x7ffa9c0ac5a0 legacy=0x7ffa9c1df200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 >> v1:172.21.15.153:6789/0 conn(0x7ffa9c0b0770 legacy=0x7ffa9c126270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 monclient: found mon.b 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c145070 con 0x7ffa9c14a700 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient: _finish_auth 0 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:06:49.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffa99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffaa3910640 5 monclient: authenticate success, global_id 4273 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.145+0000 7ffaa3910640 10 monclient: _renew_subs 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffaa3910640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffaa3910640 1 -- 172.21.15.153:0/2101296648 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9c1452c0 con 0x7ffa9c14a700 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffaa3910640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffaa3910640 10 monclient: _renew_subs 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffaa3910640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffaa3910640 1 -- 172.21.15.153:0/2101296648 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9c126b60 con 0x7ffa9c14a700 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffaa3910640 1 librados: init done 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa8c001fd0 con 0x7ffa9c14a700 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffa99ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ffa8c004fb0 con 0x7ffa9c14a700 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffa99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffa99ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffa9affd640 4 set_mon_vals no callback set 2024-03-27T03:06:49.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.146+0000 7ffa99ffb640 10 monclient: dump: 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.148+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7ffa8c01af80 con 0x7ffa9c14a700 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.149+0000 7ffaa3910640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.149+0000 7ffaa3910640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.149+0000 7ffaa3910640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:49.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.149+0000 7ffaa3910640 1 -- 172.21.15.153:0/2101296648 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa7c005080 con 0x7ffa9c14a700 2024-03-27T03:06:49.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.149+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7ffa8c084e70 con 0x7ffa9c14a700 2024-03-27T03:06:49.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.149+0000 7ffa99ffb640 10 client.4273.objecter ms_dispatch 0x7ffa9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:06:49.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.149+0000 7ffa99ffb640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:06:49.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.149+0000 7ffa99ffb640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:06:49.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.149+0000 7ffa99ffb640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-27T03:06:49.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.150+0000 7ffaa16ac640 10 monclient: get_auth_request con 0x7ffa8806dfb0 auth_method 0 2024-03-27T03:06:49.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.150+0000 7ffa99ffb640 10 client.4273.objecter ms_handle_connect 0x7ffa8806dfb0 2024-03-27T03:06:49.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.156+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa8c050750 con 0x7ffa9c14a700 2024-03-27T03:06:49.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.156+0000 7ffa99ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:49.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.156+0000 7ffa99ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:06:49.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.528+0000 7ffaa3910640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:49.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.528+0000 7ffaa3910640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:49.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.528+0000 7ffaa3910640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:06:49.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.528+0000 7ffaa3910640 1 -- 172.21.15.153:0/2101296648 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa7c005330 con 0x7ffa9c14a700 2024-03-27T03:06:49.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.529+0000 7ffa99ffb640 1 -- 172.21.15.153:0/2101296648 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7ffa8c052680 con 0x7ffa9c14a700 2024-03-27T03:06:49.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.529+0000 7ffa99ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:49.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.529+0000 7ffa99ffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:06:49.522 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:06:49.522 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:06:49.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.533+0000 7ffaa3910640 10 librados: watch_flush enter 2024-03-27T03:06:49.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.533+0000 7ffaa3910640 10 librados: watch_flush exit 2024-03-27T03:06:49.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.533+0000 7ffaa3910640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-27T03:06:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.533+0000 7ffaa3910640 10 client.4273.objecter successfully canceled tick 2024-03-27T03:06:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.533+0000 7ffaa3910640 1 -- 172.21.15.153:0/2101296648 >> v1:172.21.15.153:6816/2637937781 conn(0x7ffa8806dfb0 legacy=0x7ffa88070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.533+0000 7ffaa3910640 10 monclient: shutdown 2024-03-27T03:06:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.533+0000 7ffaa3910640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.533+0000 7ffaa3910640 1 -- 172.21.15.153:0/2101296648 >> v1:172.21.15.196:6789/0 conn(0x7ffa9c14a700 legacy=0x7ffa9c1e64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.533+0000 7ffaa3910640 1 -- 172.21.15.153:0/2101296648 shutdown_connections 2024-03-27T03:06:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.533+0000 7ffaa3910640 1 -- 172.21.15.153:0/2101296648 >> 172.21.15.153:0/2101296648 conn(0x7ffa9c06b660 msgr2=0x7ffa9c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:49.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.534+0000 7ffaa3910640 1 -- 172.21.15.153:0/2101296648 shutdown_connections 2024-03-27T03:06:49.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.534+0000 7ffaa3910640 1 -- 172.21.15.153:0/2101296648 wait complete. 2024-03-27T03:06:49.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:49.534+0000 7ffaa3910640 1 librados: shutdown 2024-03-27T03:06:49.542 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:06:56.348 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:06:56.495 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.502+0000 7fb84f50d640 10 monclient: get_monmap_and_config 2024-03-27T03:06:56.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.502+0000 7fb84f50d640 10 monclient: build_initial_monmap 2024-03-27T03:06:56.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.502+0000 7fb84f50d640 10 monclient: monmap: 2024-03-27T03:06:56.496 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:56.496 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:56.496 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:06:56.503821+0000 2024-03-27T03:06:56.496 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:06:56.503821+0000 2024-03-27T03:06:56.496 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:56.496 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:56.496 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:56.496 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:06:56.496 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:06:56.496 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:56.497 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.503+0000 7fb84f50d640 1 Processor -- start 2024-03-27T03:06:56.497 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.504+0000 7fb84f50d640 1 -- start start 2024-03-27T03:06:56.497 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.504+0000 7fb84f50d640 10 monclient: init 2024-03-27T03:06:56.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.504+0000 7fb84f50d640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:56.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.504+0000 7fb84f50d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:06:56.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.504+0000 7fb84f50d640 10 monclient(hunting): picked mon.noname-b con 0x7fb848071de0 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:56.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.504+0000 7fb84f50d640 10 monclient(hunting): picked mon.noname-c con 0x7fb8480721c0 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:56.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.504+0000 7fb84f50d640 10 monclient(hunting): picked mon.noname-a con 0x7fb848145ae0 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:56.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.505+0000 7fb84f50d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb848145650 con 0x7fb848145ae0 2024-03-27T03:06:56.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.505+0000 7fb84f50d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8480ae260 con 0x7fb8480721c0 2024-03-27T03:06:56.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.505+0000 7fb84f50d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8481420d0 con 0x7fb848071de0 2024-03-27T03:06:56.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.505+0000 7fb84f50d640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:56.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.505+0000 7fb84f50d640 10 monclient(hunting): authenticate will time out at 1376.810613s 2024-03-27T03:06:56.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.507+0000 7fb84daaa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb848145ae0 0x7fb848145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:51879/0 (socket says 172.21.15.153:51879) 2024-03-27T03:06:56.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.507+0000 7fb84daaa640 1 -- 172.21.15.153:0/3839707732 learned_addr learned my addr 172.21.15.153:0/3839707732 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:56.510 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 1 -- 172.21.15.153:0/3839707732 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568224612 0 0) 0x7fb848145650 con 0x7fb848145ae0 2024-03-27T03:06:56.510 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient(hunting): _negotiate 2024-03-27T03:06:56.510 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:56.510 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:56.510 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient(hunting): my global_id is 4266 2024-03-27T03:06:56.510 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 1 -- 172.21.15.153:0/3839707732 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb834003520 con 0x7fb848145ae0 2024-03-27T03:06:56.510 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 1 -- 172.21.15.153:0/3839707732 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504433292 0 0) 0x7fb8481420d0 con 0x7fb848071de0 2024-03-27T03:06:56.510 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient(hunting): _negotiate 2024-03-27T03:06:56.510 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:56.510 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient(hunting): my global_id is 4289 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 1 -- 172.21.15.153:0/3839707732 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb848145650 con 0x7fb848071de0 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 1 -- 172.21.15.153:0/3839707732 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2757245632 0 0) 0x7fb848145650 con 0x7fb848071de0 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 1 -- 172.21.15.153:0/3839707732 >> v1:172.21.15.196:6789/0 conn(0x7fb8480721c0 legacy=0x7fb84813f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 1 -- 172.21.15.153:0/3839707732 >> v1:172.21.15.153:6789/0 conn(0x7fb848145ae0 legacy=0x7fb848145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient: _finish_hunting 0 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 1 monclient: found mon.noname-b 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 1 -- 172.21.15.153:0/3839707732 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8480b0590 con 0x7fb848071de0 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient: _finish_auth 0 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient: _check_auth_tickets 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 1 -- 172.21.15.153:0/3839707732 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fb844001520 con 0x7fb848071de0 2024-03-27T03:06:56.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient: dump: 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 1 -- 172.21.15.153:0/3839707732 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb844001700 con 0x7fb848071de0 2024-03-27T03:06:56.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 1 -- 172.21.15.153:0/3839707732 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fb844001520 con 0x7fb848071de0 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb83ffff640 10 monclient: dump: 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb84f50d640 5 monclient: authenticate success, global_id 4289 2024-03-27T03:06:56.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb84f50d640 10 monclient: get_monmap_and_config success 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.509+0000 7fb84f50d640 4 set_mon_vals no callback set 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.510+0000 7fb84f50d640 10 monclient: shutdown 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.510+0000 7fb84f50d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.510+0000 7fb84f50d640 1 -- 172.21.15.153:0/3839707732 >> v1:172.21.15.153:6790/0 conn(0x7fb848071de0 legacy=0x7fb8480aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.511+0000 7fb84f50d640 1 -- 172.21.15.153:0/3839707732 shutdown_connections 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.511+0000 7fb84f50d640 1 -- 172.21.15.153:0/3839707732 >> 172.21.15.153:0/3839707732 conn(0x7fb84806b660 msgr2=0x7fb84806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.511+0000 7fb84f50d640 1 -- 172.21.15.153:0/3839707732 shutdown_connections 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.512+0000 7fb84f50d640 1 -- 172.21.15.153:0/3839707732 wait complete. 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.513+0000 7fb84f50d640 10 monclient: build_initial_monmap 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.513+0000 7fb84f50d640 10 monclient: monmap: 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:06:56.514 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.513+0000 7fb84f50d640 1 librados: starting msgr at 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.513+0000 7fb84f50d640 1 librados: starting objecter 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.513+0000 7fb84f50d640 1 Processor -- start 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.513+0000 7fb84f50d640 1 -- start start 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.513+0000 7fb84f50d640 1 librados: setting wanted keys 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.513+0000 7fb84f50d640 1 librados: calling monclient init 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.513+0000 7fb84f50d640 10 monclient: init 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.514+0000 7fb84f50d640 10 monclient: _reopen_session rank -1 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.515+0000 7fb84f50d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.515+0000 7fb84f50d640 10 monclient(hunting): picked mon.noname-c con 0x7fb848071de0 addr v1:172.21.15.153:6790/0 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.515+0000 7fb84f50d640 10 monclient(hunting): picked mon.noname-a con 0x7fb848145ae0 addr v1:172.21.15.153:6789/0 2024-03-27T03:06:56.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.515+0000 7fb84f50d640 10 monclient(hunting): picked mon.noname-b con 0x7fb84811c380 addr v1:172.21.15.196:6789/0 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.515+0000 7fb84f50d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84811d5f0 con 0x7fb848145ae0 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.515+0000 7fb84f50d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84811c810 con 0x7fb84811c380 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.515+0000 7fb84f50d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8481d2f90 con 0x7fb848071de0 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.515+0000 7fb84f50d640 10 monclient(hunting): _renew_subs 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.515+0000 7fb84f50d640 10 monclient(hunting): authenticate will time out at 1376.821100s 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.516+0000 7fb84daaa640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fb84811c380 0x7fb8481dd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:37250/0 (socket says 172.21.15.153:37250) 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.516+0000 7fb84daaa640 1 -- 172.21.15.153:0/4091721775 learned_addr learned my addr 172.21.15.153:0/4091721775 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.516+0000 7fb83dffb640 10 client.?.objecter ms_handle_connect 0x7fb84811c380 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.516+0000 7fb83dffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.517+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122981555 0 0) 0x7fb84811c810 con 0x7fb84811c380 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.517+0000 7fb83dffb640 10 monclient(hunting): _negotiate 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.517+0000 7fb83dffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.517+0000 7fb83dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.517+0000 7fb83dffb640 10 monclient(hunting): my global_id is 4279 2024-03-27T03:06:56.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.517+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c004eb0 con 0x7fb84811c380 2024-03-27T03:06:56.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.518+0000 7fb83dffb640 10 client.?.objecter ms_handle_connect 0x7fb848145ae0 2024-03-27T03:06:56.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.518+0000 7fb83dffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:56.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.518+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951880228 0 0) 0x7fb84811d5f0 con 0x7fb848145ae0 2024-03-27T03:06:56.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.518+0000 7fb83dffb640 10 monclient(hunting): _negotiate 2024-03-27T03:06:56.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.518+0000 7fb83dffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:06:56.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.518+0000 7fb83dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:06:56.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.518+0000 7fb83dffb640 10 monclient(hunting): my global_id is 4269 2024-03-27T03:06:56.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.518+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84811c810 con 0x7fb848145ae0 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2593557672 0 0) 0x7fb84811c810 con 0x7fb848145ae0 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb84811d5f0 con 0x7fb848145ae0 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fb8380018a0 con 0x7fb848145ae0 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 10 monclient(hunting): dump: 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:56.523 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 10 client.?.objecter ms_handle_connect 0x7fb848071de0 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 61432906 0 0) 0x7fb84811d5f0 con 0x7fb848145ae0 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 >> v1:172.21.15.153:6790/0 conn(0x7fb848071de0 legacy=0x7fb8481d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 >> v1:172.21.15.196:6789/0 conn(0x7fb84811c380 legacy=0x7fb8481dd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 10 monclient: _finish_hunting 0 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 1 monclient: found mon.a 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84811b200 con 0x7fb848145ae0 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 10 monclient: _finish_auth 0 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 10 monclient: _check_auth_tickets 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.519+0000 7fb83dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb84f50d640 5 monclient: authenticate success, global_id 4269 2024-03-27T03:06:56.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb84f50d640 10 monclient: _renew_subs 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb84f50d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb84f50d640 1 -- 172.21.15.153:0/4091721775 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84813e040 con 0x7fb848145ae0 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb84f50d640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb84f50d640 10 monclient: _renew_subs 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb84f50d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb84f50d640 1 -- 172.21.15.153:0/4091721775 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8481d3da0 con 0x7fb848145ae0 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb84f50d640 1 librados: init done 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8380035f0 con 0x7fb848145ae0 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb83dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fb8380039c0 con 0x7fb848145ae0 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb83dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb83dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb83f7fe640 4 set_mon_vals no callback set 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.520+0000 7fb83dffb640 10 monclient: dump: 2024-03-27T03:06:56.525 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.523+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fb838003ba0 con 0x7fb848145ae0 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.523+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fb838083820 con 0x7fb848145ae0 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.524+0000 7fb83dffb640 10 client.4269.objecter ms_dispatch 0x7fb848000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.524+0000 7fb83dffb640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.524+0000 7fb83dffb640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:06:56.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.524+0000 7fb83dffb640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-27T03:06:56.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.524+0000 7fb84d2a9640 10 monclient: get_auth_request con 0x7fb82c06dd40 auth_method 0 2024-03-27T03:06:56.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.524+0000 7fb83dffb640 10 client.4269.objecter ms_handle_connect 0x7fb82c06dd40 2024-03-27T03:06:56.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.525+0000 7fb84f50d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:56.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.525+0000 7fb84f50d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:56.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.525+0000 7fb84f50d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:06:56.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.525+0000 7fb84f50d640 1 -- 172.21.15.153:0/4091721775 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb828005080 con 0x7fb848145ae0 2024-03-27T03:06:56.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.532+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb83804f100 con 0x7fb848145ae0 2024-03-27T03:06:56.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.532+0000 7fb83dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:06:56.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.532+0000 7fb83dffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:06:56.843 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.850+0000 7fb84f50d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:56.843 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.850+0000 7fb84f50d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:56.843 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.850+0000 7fb84f50d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:06:56.843 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.850+0000 7fb84f50d640 1 -- 172.21.15.153:0/4091721775 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb828005330 con 0x7fb848145ae0 2024-03-27T03:06:56.845 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.852+0000 7fb83dffb640 1 -- 172.21.15.153:0/4091721775 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fb838051030 con 0x7fb848145ae0 2024-03-27T03:06:56.845 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.852+0000 7fb83dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:06:56.845 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.852+0000 7fb83dffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:06:56.845 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:06:56.845 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:06:56.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.855+0000 7fb84f50d640 10 librados: watch_flush enter 2024-03-27T03:06:56.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.856+0000 7fb84f50d640 10 librados: watch_flush exit 2024-03-27T03:06:56.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.856+0000 7fb84f50d640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-27T03:06:56.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.856+0000 7fb84f50d640 10 client.4269.objecter successfully canceled tick 2024-03-27T03:06:56.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.856+0000 7fb84f50d640 1 -- 172.21.15.153:0/4091721775 >> v1:172.21.15.153:6816/2637937781 conn(0x7fb82c06dd40 legacy=0x7fb82c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:56.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.856+0000 7fb84f50d640 10 monclient: shutdown 2024-03-27T03:06:56.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.856+0000 7fb84f50d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:06:56.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.856+0000 7fb84f50d640 1 -- 172.21.15.153:0/4091721775 >> v1:172.21.15.153:6789/0 conn(0x7fb848145ae0 legacy=0x7fb84811bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:06:56.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.856+0000 7fb84f50d640 1 -- 172.21.15.153:0/4091721775 shutdown_connections 2024-03-27T03:06:56.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.856+0000 7fb84f50d640 1 -- 172.21.15.153:0/4091721775 >> 172.21.15.153:0/4091721775 conn(0x7fb84806b660 msgr2=0x7fb84806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:06:56.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.856+0000 7fb84f50d640 1 -- 172.21.15.153:0/4091721775 shutdown_connections 2024-03-27T03:06:56.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.856+0000 7fb84f50d640 1 -- 172.21.15.153:0/4091721775 wait complete. 2024-03-27T03:06:56.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:06:56.857+0000 7fb84f50d640 1 librados: shutdown 2024-03-27T03:06:56.866 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:07:03.672 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:07:03.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.825+0000 7fe9abfff640 10 monclient: get_monmap_and_config 2024-03-27T03:07:03.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.825+0000 7fe9abfff640 10 monclient: build_initial_monmap 2024-03-27T03:07:03.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.825+0000 7fe9abfff640 10 monclient: monmap: 2024-03-27T03:07:03.820 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:03.820 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:03.820 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:07:03.827872+0000 2024-03-27T03:07:03.820 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:07:03.827872+0000 2024-03-27T03:07:03.820 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:03.820 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:03.820 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:03.820 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:07:03.820 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:07:03.820 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:03.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.826+0000 7fe9abfff640 1 Processor -- start 2024-03-27T03:07:03.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.826+0000 7fe9abfff640 1 -- start start 2024-03-27T03:07:03.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.826+0000 7fe9abfff640 10 monclient: init 2024-03-27T03:07:03.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.827+0000 7fe9abfff640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:03.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.827+0000 7fe9abfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:07:03.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.827+0000 7fe9abfff640 10 monclient(hunting): picked mon.noname-a con 0x7fe9ac0b2800 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:03.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.827+0000 7fe9abfff640 10 monclient(hunting): picked mon.noname-c con 0x7fe9ac0b3220 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:03.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.827+0000 7fe9abfff640 10 monclient(hunting): picked mon.noname-b con 0x7fe9ac1439c0 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:03.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.827+0000 7fe9abfff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac0b3840 con 0x7fe9ac0b2800 2024-03-27T03:07:03.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.827+0000 7fe9abfff640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac06f230 con 0x7fe9ac0b3220 2024-03-27T03:07:03.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.827+0000 7fe9abfff640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac06ef50 con 0x7fe9ac1439c0 2024-03-27T03:07:03.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.827+0000 7fe9abfff640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:03.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.827+0000 7fe9abfff640 10 monclient(hunting): authenticate will time out at 1384.134281s 2024-03-27T03:07:03.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9ab7fe640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fe9ac1439c0 0x7fe9ac151c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:59563/0 (socket says 172.21.15.153:59563) 2024-03-27T03:07:03.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9ab7fe640 1 -- 172.21.15.153:0/606345418 learned_addr learned my addr 172.21.15.153:0/606345418 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:03.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 1 -- 172.21.15.153:0/606345418 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679242330 0 0) 0x7fe9ac06ef50 con 0x7fe9ac1439c0 2024-03-27T03:07:03.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient(hunting): my global_id is 4292 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 1 -- 172.21.15.153:0/606345418 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe998003520 con 0x7fe9ac1439c0 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 1 -- 172.21.15.153:0/606345418 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241962861 0 0) 0x7fe9ac0b3840 con 0x7fe9ac0b2800 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient(hunting): my global_id is 4272 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 1 -- 172.21.15.153:0/606345418 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac06ef50 con 0x7fe9ac0b2800 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 1 -- 172.21.15.153:0/606345418 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2254488945 0 0) 0x7fe998003520 con 0x7fe9ac1439c0 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 1 -- 172.21.15.153:0/606345418 >> v1:172.21.15.196:6789/0 conn(0x7fe9ac0b3220 legacy=0x7fe9ac146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 1 -- 172.21.15.153:0/606345418 >> v1:172.21.15.153:6789/0 conn(0x7fe9ac0b2800 legacy=0x7fe9ac1432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:03.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:07:03.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 1 monclient: found mon.noname-b 2024-03-27T03:07:03.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:03.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:07:03.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 1 -- 172.21.15.153:0/606345418 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac071d30 con 0x7fe9ac1439c0 2024-03-27T03:07:03.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient: _finish_auth 0 2024-03-27T03:07:03.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:07:03.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:03.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 1 -- 172.21.15.153:0/606345418 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe9a4001a10 con 0x7fe9ac1439c0 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9aa7fc640 10 monclient: dump: 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:03.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9abfff640 5 monclient: authenticate success, global_id 4292 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.830+0000 7fe9abfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.831+0000 7fe9aa7fc640 1 -- 172.21.15.153:0/606345418 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9a4001b70 con 0x7fe9ac1439c0 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.831+0000 7fe9aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.831+0000 7fe9aa7fc640 1 -- 172.21.15.153:0/606345418 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe9a4001f80 con 0x7fe9ac1439c0 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.831+0000 7fe9aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.831+0000 7fe9aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.831+0000 7fe9aa7fc640 10 monclient: dump: 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:03.828 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:03.829 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:03.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.831+0000 7fe9abfff640 10 monclient: get_monmap_and_config success 2024-03-27T03:07:03.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.831+0000 7fe9abfff640 4 set_mon_vals no callback set 2024-03-27T03:07:03.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.832+0000 7fe9abfff640 10 monclient: shutdown 2024-03-27T03:07:03.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.832+0000 7fe9abfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:03.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.832+0000 7fe9abfff640 1 -- 172.21.15.153:0/606345418 >> v1:172.21.15.153:6790/0 conn(0x7fe9ac1439c0 legacy=0x7fe9ac151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:03.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.832+0000 7fe9abfff640 1 -- 172.21.15.153:0/606345418 shutdown_connections 2024-03-27T03:07:03.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.832+0000 7fe9abfff640 1 -- 172.21.15.153:0/606345418 >> 172.21.15.153:0/606345418 conn(0x7fe9ac066a00 msgr2=0x7fe9ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:03.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.833+0000 7fe9abfff640 1 -- 172.21.15.153:0/606345418 shutdown_connections 2024-03-27T03:07:03.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.835+0000 7fe9abfff640 1 -- 172.21.15.153:0/606345418 wait complete. 2024-03-27T03:07:03.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.835+0000 7fe9abfff640 10 monclient: build_initial_monmap 2024-03-27T03:07:03.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.835+0000 7fe9abfff640 10 monclient: monmap: 2024-03-27T03:07:03.831 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.836+0000 7fe9abfff640 1 librados: starting msgr at 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.836+0000 7fe9abfff640 1 librados: starting objecter 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.836+0000 7fe9abfff640 1 Processor -- start 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.836+0000 7fe9abfff640 1 -- start start 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.836+0000 7fe9abfff640 1 librados: setting wanted keys 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.836+0000 7fe9abfff640 1 librados: calling monclient init 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.836+0000 7fe9abfff640 10 monclient: init 2024-03-27T03:07:03.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.837+0000 7fe9abfff640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:03.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.837+0000 7fe9abfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:07:03.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.837+0000 7fe9abfff640 10 monclient(hunting): picked mon.noname-c con 0x7fe9ac0b2800 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:03.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.837+0000 7fe9abfff640 10 monclient(hunting): picked mon.noname-a con 0x7fe9ac0b3220 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:03.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.837+0000 7fe9abfff640 10 monclient(hunting): picked mon.noname-b con 0x7fe9ac1439c0 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:03.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.837+0000 7fe9abfff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac13e8e0 con 0x7fe9ac0b3220 2024-03-27T03:07:03.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.837+0000 7fe9abfff640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac13f9a0 con 0x7fe9ac1439c0 2024-03-27T03:07:03.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.837+0000 7fe9abfff640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac147640 con 0x7fe9ac0b2800 2024-03-27T03:07:03.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.837+0000 7fe9abfff640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:03.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.837+0000 7fe9abfff640 10 monclient(hunting): authenticate will time out at 1384.144071s 2024-03-27T03:07:03.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.840+0000 7fe9ab7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe9ac1439c0 0x7fe9ac1e60f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:35594/0 (socket says 172.21.15.153:35594) 2024-03-27T03:07:03.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.840+0000 7fe9ab7fe640 1 -- 172.21.15.153:0/3918293396 learned_addr learned my addr 172.21.15.153:0/3918293396 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:03.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.840+0000 7fe9a3fff640 10 client.?.objecter ms_handle_connect 0x7fe9ac1439c0 2024-03-27T03:07:03.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.840+0000 7fe9a3fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:03.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.841+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014597188 0 0) 0x7fe9ac13f9a0 con 0x7fe9ac1439c0 2024-03-27T03:07:03.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.841+0000 7fe9a3fff640 10 monclient(hunting): _negotiate 2024-03-27T03:07:03.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.841+0000 7fe9a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:03.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.841+0000 7fe9a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:03.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.841+0000 7fe9a3fff640 10 monclient(hunting): my global_id is 4282 2024-03-27T03:07:03.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.841+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98c004eb0 con 0x7fe9ac1439c0 2024-03-27T03:07:03.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.841+0000 7fe9a3fff640 10 client.?.objecter ms_handle_connect 0x7fe9ac0b3220 2024-03-27T03:07:03.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.841+0000 7fe9a3fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:03.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 client.?.objecter ms_handle_connect 0x7fe9ac0b2800 2024-03-27T03:07:03.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:03.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1188724900 0 0) 0x7fe98c004eb0 con 0x7fe9ac1439c0 2024-03-27T03:07:03.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9ac13f9a0 con 0x7fe9ac1439c0 2024-03-27T03:07:03.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331633820 0 0) 0x7fe9ac13e8e0 con 0x7fe9ac0b3220 2024-03-27T03:07:03.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 monclient(hunting): _negotiate 2024-03-27T03:07:03.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:03.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:03.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 monclient(hunting): my global_id is 4275 2024-03-27T03:07:03.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98c004eb0 con 0x7fe9ac0b3220 2024-03-27T03:07:03.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe9a4001f40 con 0x7fe9ac1439c0 2024-03-27T03:07:03.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:03.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:07:03.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 monclient(hunting): dump: 2024-03-27T03:07:03.840 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:03.840 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:03.840 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:03.840 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:03.840 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:03.840 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:03.840 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:03.840 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030614775 0 0) 0x7fe9ac147640 con 0x7fe9ac0b2800 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 monclient(hunting): _negotiate 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 10 monclient(hunting): my global_id is 4295 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.842+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac13e8e0 con 0x7fe9ac0b2800 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.843+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4086776477 0 0) 0x7fe98c004eb0 con 0x7fe9ac0b3220 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.844+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9ac147640 con 0x7fe9ac0b3220 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 850901085 0 0) 0x7fe9ac13e8e0 con 0x7fe9ac0b2800 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe98c004eb0 con 0x7fe9ac0b2800 2024-03-27T03:07:03.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe99c001730 con 0x7fe9ac0b3220 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 10 monclient(hunting): dump: 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:03.842 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe9940018b0 con 0x7fe9ac0b2800 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 10 monclient(hunting): dump: 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:03.843 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3541379046 0 0) 0x7fe9ac13f9a0 con 0x7fe9ac1439c0 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 >> v1:172.21.15.153:6790/0 conn(0x7fe9ac0b2800 legacy=0x7fe9ac13db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 >> v1:172.21.15.153:6789/0 conn(0x7fe9ac0b3220 legacy=0x7fe9ac06fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 10 monclient: _finish_hunting 0 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 1 monclient: found mon.b 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac13ed70 con 0x7fe9ac1439c0 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 10 monclient: _finish_auth 0 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 10 monclient: _check_auth_tickets 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9abfff640 5 monclient: authenticate success, global_id 4282 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9abfff640 10 monclient: _renew_subs 2024-03-27T03:07:03.844 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:03.845 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9abfff640 1 -- 172.21.15.153:0/3918293396 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ac13e330 con 0x7fe9ac1439c0 2024-03-27T03:07:03.845 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9abfff640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:07:03.845 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9abfff640 10 monclient: _renew_subs 2024-03-27T03:07:03.845 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:03.845 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9abfff640 1 -- 172.21.15.153:0/3918293396 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ac0705a0 con 0x7fe9ac1439c0 2024-03-27T03:07:03.845 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.845+0000 7fe9abfff640 1 librados: init done 2024-03-27T03:07:03.846 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.846+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9a4004e80 con 0x7fe9ac1439c0 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.846+0000 7fe9a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.846+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe9a4005250 con 0x7fe9ac1439c0 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.846+0000 7fe9a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.846+0000 7fe9a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.846+0000 7fe9a3fff640 10 monclient: dump: 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:03.847 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.846+0000 7fe9abfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.846+0000 7fe9a9ffb640 4 set_mon_vals no callback set 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.847+0000 7fe9abfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.847+0000 7fe9abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.847+0000 7fe9abfff640 1 -- 172.21.15.153:0/3918293396 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ac04f730 con 0x7fe9ac1439c0 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.848+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fe9a40053b0 con 0x7fe9ac1439c0 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.849+0000 7fe9aaffd640 10 monclient: get_auth_request con 0x7fe98c06e2c0 auth_method 0 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.849+0000 7fe9a3fff640 10 client.4282.objecter ms_handle_connect 0x7fe98c06e2c0 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.850+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fe9a4084e80 con 0x7fe9ac1439c0 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.850+0000 7fe9a3fff640 10 client.4282.objecter ms_dispatch 0x7fe9ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.850+0000 7fe9a3fff640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.850+0000 7fe9a3fff640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:07:03.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.850+0000 7fe9a3fff640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-27T03:07:03.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.855+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9a4050760 con 0x7fe9ac1439c0 2024-03-27T03:07:03.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.855+0000 7fe9a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:03.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:03.855+0000 7fe9a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:07:04.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.177+0000 7fe9abfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:04.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.177+0000 7fe9abfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:04.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.177+0000 7fe9abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:04.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.177+0000 7fe9abfff640 1 -- 172.21.15.153:0/3918293396 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9ac145920 con 0x7fe9ac1439c0 2024-03-27T03:07:04.172 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:07:04.172 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:07:04.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.178+0000 7fe9a3fff640 1 -- 172.21.15.153:0/3918293396 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fe9a4052690 con 0x7fe9ac1439c0 2024-03-27T03:07:04.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.178+0000 7fe9a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:04.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.178+0000 7fe9a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:07:04.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.182+0000 7fe9abfff640 10 librados: watch_flush enter 2024-03-27T03:07:04.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.182+0000 7fe9abfff640 10 librados: watch_flush exit 2024-03-27T03:07:04.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.182+0000 7fe9abfff640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-27T03:07:04.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.182+0000 7fe9abfff640 10 client.4282.objecter successfully canceled tick 2024-03-27T03:07:04.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.183+0000 7fe9abfff640 1 -- 172.21.15.153:0/3918293396 >> v1:172.21.15.153:6816/2637937781 conn(0x7fe98c06e2c0 legacy=0x7fe98c070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:04.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.183+0000 7fe9abfff640 10 monclient: shutdown 2024-03-27T03:07:04.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.183+0000 7fe9abfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:04.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.183+0000 7fe9abfff640 1 -- 172.21.15.153:0/3918293396 >> v1:172.21.15.196:6789/0 conn(0x7fe9ac1439c0 legacy=0x7fe9ac1e60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:04.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.183+0000 7fe9abfff640 1 -- 172.21.15.153:0/3918293396 shutdown_connections 2024-03-27T03:07:04.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.183+0000 7fe9abfff640 1 -- 172.21.15.153:0/3918293396 >> 172.21.15.153:0/3918293396 conn(0x7fe9ac066a00 msgr2=0x7fe9ac140590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:04.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.183+0000 7fe9abfff640 1 -- 172.21.15.153:0/3918293396 shutdown_connections 2024-03-27T03:07:04.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.184+0000 7fe9abfff640 1 -- 172.21.15.153:0/3918293396 wait complete. 2024-03-27T03:07:04.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:04.184+0000 7fe9abfff640 1 librados: shutdown 2024-03-27T03:07:04.195 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:07:11.002 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:07:11.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.158+0000 7f70a55d6640 10 monclient: get_monmap_and_config 2024-03-27T03:07:11.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.158+0000 7f70a55d6640 10 monclient: build_initial_monmap 2024-03-27T03:07:11.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.158+0000 7f70a55d6640 10 monclient: monmap: 2024-03-27T03:07:11.152 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:11.152 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:11.152 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:07:11.159383+0000 2024-03-27T03:07:11.152 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:07:11.159383+0000 2024-03-27T03:07:11.152 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:11.152 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:11.153 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:11.153 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:07:11.153 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:07:11.153 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:11.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.160+0000 7f70a55d6640 1 Processor -- start 2024-03-27T03:07:11.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.160+0000 7f70a55d6640 1 -- start start 2024-03-27T03:07:11.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.160+0000 7f70a55d6640 10 monclient: init 2024-03-27T03:07:11.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.161+0000 7f70a55d6640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:11.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.161+0000 7f70a55d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:07:11.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.161+0000 7f70a55d6640 10 monclient(hunting): picked mon.noname-b con 0x7f70a00a9ac0 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:11.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.161+0000 7f70a55d6640 10 monclient(hunting): picked mon.noname-c con 0x7f70a013ee90 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:11.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.161+0000 7f70a55d6640 10 monclient(hunting): picked mon.noname-a con 0x7f70a00af900 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:11.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.161+0000 7f70a55d6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a00afd10 con 0x7f70a00af900 2024-03-27T03:07:11.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.161+0000 7f70a55d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a00af6f0 con 0x7f70a013ee90 2024-03-27T03:07:11.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.161+0000 7f70a55d6640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a00aac80 con 0x7f70a00a9ac0 2024-03-27T03:07:11.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.161+0000 7f70a55d6640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:11.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.161+0000 7f70a55d6640 10 monclient(hunting): authenticate will time out at 1391.466750s 2024-03-27T03:07:11.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.164+0000 7f709f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f70a00af900 0x7f70a0149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:53255/0 (socket says 172.21.15.153:53255) 2024-03-27T03:07:11.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.164+0000 7f709f7fe640 1 -- 172.21.15.153:0/3319093096 learned_addr learned my addr 172.21.15.153:0/3319093096 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:11.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.164+0000 7f709dffb640 1 -- 172.21.15.153:0/3319093096 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931593311 0 0) 0x7f70a00afd10 con 0x7f70a00af900 2024-03-27T03:07:11.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.164+0000 7f709dffb640 10 monclient(hunting): _negotiate 2024-03-27T03:07:11.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.164+0000 7f709dffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:11.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.164+0000 7f709dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:11.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.164+0000 7f709dffb640 10 monclient(hunting): my global_id is 4278 2024-03-27T03:07:11.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 1 -- 172.21.15.153:0/3319093096 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7088003520 con 0x7f70a00af900 2024-03-27T03:07:11.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 1 -- 172.21.15.153:0/3319093096 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588529749 0 0) 0x7f70a00af6f0 con 0x7f70a013ee90 2024-03-27T03:07:11.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 10 monclient(hunting): _negotiate 2024-03-27T03:07:11.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:11.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:11.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 10 monclient(hunting): my global_id is 4285 2024-03-27T03:07:11.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 1 -- 172.21.15.153:0/3319093096 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a00afd10 con 0x7f70a013ee90 2024-03-27T03:07:11.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 1 -- 172.21.15.153:0/3319093096 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 556402158 0 0) 0x7f70a00afd10 con 0x7f70a013ee90 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 1 -- 172.21.15.153:0/3319093096 >> v1:172.21.15.153:6790/0 conn(0x7f70a00a9ac0 legacy=0x7f70a00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 1 -- 172.21.15.153:0/3319093096 >> v1:172.21.15.153:6789/0 conn(0x7f70a00af900 legacy=0x7f70a0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 10 monclient: _finish_hunting 0 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 1 monclient: found mon.noname-c 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 1 -- 172.21.15.153:0/3319093096 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a00aaf60 con 0x7f70a013ee90 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 10 monclient: _finish_auth 0 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 10 monclient: _check_auth_tickets 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f70a55d6640 5 monclient: authenticate success, global_id 4285 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f70a55d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 1 -- 172.21.15.153:0/3319093096 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f7094001670 con 0x7f70a013ee90 2024-03-27T03:07:11.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f709dffb640 10 monclient: dump: 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.165+0000 7f70a55d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.166+0000 7f709dffb640 1 -- 172.21.15.153:0/3319093096 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7094001890 con 0x7f70a013ee90 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.166+0000 7f709dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:11.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.166+0000 7f709dffb640 1 -- 172.21.15.153:0/3319093096 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f7094001c20 con 0x7f70a013ee90 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.166+0000 7f709dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.166+0000 7f709dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.166+0000 7f709dffb640 10 monclient: dump: 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.166+0000 7f70a55d6640 10 monclient: get_monmap_and_config success 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.166+0000 7f70a55d6640 4 set_mon_vals no callback set 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.167+0000 7f70a55d6640 10 monclient: shutdown 2024-03-27T03:07:11.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.167+0000 7f70a55d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:11.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.167+0000 7f70a55d6640 1 -- 172.21.15.153:0/3319093096 >> v1:172.21.15.196:6789/0 conn(0x7f70a013ee90 legacy=0x7f70a013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:11.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.167+0000 7f70a55d6640 1 -- 172.21.15.153:0/3319093096 shutdown_connections 2024-03-27T03:07:11.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.167+0000 7f70a55d6640 1 -- 172.21.15.153:0/3319093096 >> 172.21.15.153:0/3319093096 conn(0x7f70a00b35e0 msgr2=0x7f70a00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:11.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.167+0000 7f70a55d6640 1 -- 172.21.15.153:0/3319093096 shutdown_connections 2024-03-27T03:07:11.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.167+0000 7f70a55d6640 1 -- 172.21.15.153:0/3319093096 wait complete. 2024-03-27T03:07:11.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.168+0000 7f70a55d6640 10 monclient: build_initial_monmap 2024-03-27T03:07:11.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.168+0000 7f70a55d6640 10 monclient: monmap: 2024-03-27T03:07:11.165 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:11.165 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:07:11.165 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:07:11.165 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:07:11.165 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:11.165 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:11.165 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:11.165 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.168+0000 7f70a55d6640 1 librados: starting msgr at 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.168+0000 7f70a55d6640 1 librados: starting objecter 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.168+0000 7f70a55d6640 1 Processor -- start 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 1 -- start start 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 1 librados: setting wanted keys 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 1 librados: calling monclient init 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 10 monclient: init 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 10 monclient(hunting): picked mon.noname-c con 0x7f70a00a9ac0 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 10 monclient(hunting): picked mon.noname-a con 0x7f70a00af900 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:11.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 10 monclient(hunting): picked mon.noname-b con 0x7f70a0140850 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:11.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a013ff40 con 0x7f70a00af900 2024-03-27T03:07:11.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a0142000 con 0x7f70a0140850 2024-03-27T03:07:11.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a0142590 con 0x7f70a00a9ac0 2024-03-27T03:07:11.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:11.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.169+0000 7f70a55d6640 10 monclient(hunting): authenticate will time out at 1391.475464s 2024-03-27T03:07:11.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.173+0000 7f709f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f70a0140850 0x7f70a01d9d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:35916/0 (socket says 172.21.15.153:35916) 2024-03-27T03:07:11.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.173+0000 7f709f7fe640 1 -- 172.21.15.153:0/1427557835 learned_addr learned my addr 172.21.15.153:0/1427557835 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:11.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 client.?.objecter ms_handle_connect 0x7f70a0140850 2024-03-27T03:07:11.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:11.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 client.?.objecter ms_handle_connect 0x7f70a00af900 2024-03-27T03:07:11.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:11.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642179852 0 0) 0x7f70a013ff40 con 0x7f70a00af900 2024-03-27T03:07:11.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 monclient(hunting): _negotiate 2024-03-27T03:07:11.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:11.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 monclient(hunting): my global_id is 4281 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f708c004eb0 con 0x7f70a00af900 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683526931 0 0) 0x7f70a0142000 con 0x7f70a0140850 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 monclient(hunting): _negotiate 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 monclient(hunting): my global_id is 4288 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a013ff40 con 0x7f70a0140850 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1749375934 0 0) 0x7f708c004eb0 con 0x7f70a00af900 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70a0142000 con 0x7f70a00af900 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f70940015b0 con 0x7f70a00af900 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:07:11.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 monclient(hunting): dump: 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.174+0000 7f70877fe640 10 client.?.objecter ms_handle_connect 0x7f70a00a9ac0 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:11.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1393428039 0 0) 0x7f70a013ff40 con 0x7f70a0140850 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f708c004eb0 con 0x7f70a0140850 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f7098001b50 con 0x7f70a0140850 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 10 monclient(hunting): dump: 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:11.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173560668 0 0) 0x7f70a0142590 con 0x7f70a00a9ac0 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 10 monclient(hunting): _negotiate 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 10 monclient(hunting): my global_id is 4301 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a013ff40 con 0x7f70a00a9ac0 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2966543457 0 0) 0x7f70a0142000 con 0x7f70a00af900 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 >> v1:172.21.15.153:6790/0 conn(0x7f70a00a9ac0 legacy=0x7f70a013f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 >> v1:172.21.15.196:6789/0 conn(0x7f70a0140850 legacy=0x7f70a01d9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 10 monclient: _finish_hunting 0 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 1 monclient: found mon.a 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a0072480 con 0x7f70a00af900 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 10 monclient: _finish_auth 0 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 10 monclient: _check_auth_tickets 2024-03-27T03:07:11.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:11.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70a55d6640 5 monclient: authenticate success, global_id 4281 2024-03-27T03:07:11.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70a55d6640 10 monclient: _renew_subs 2024-03-27T03:07:11.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70a55d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:07:11.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.175+0000 7f70a55d6640 1 -- 172.21.15.153:0/1427557835 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a01e49b0 con 0x7f70a00af900 2024-03-27T03:07:11.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.176+0000 7f70a55d6640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:07:11.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.176+0000 7f70a55d6640 10 monclient: _renew_subs 2024-03-27T03:07:11.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.176+0000 7f70a55d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:07:11.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.176+0000 7f70a55d6640 1 -- 172.21.15.153:0/1427557835 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a01da680 con 0x7f70a00af900 2024-03-27T03:07:11.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.176+0000 7f70a55d6640 1 librados: init done 2024-03-27T03:07:11.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.178+0000 7f70a55d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:11.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.178+0000 7f70a55d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:11.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.178+0000 7f70a55d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:07:11.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.178+0000 7f70a55d6640 1 -- 172.21.15.153:0/1427557835 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a01da930 con 0x7f70a00af900 2024-03-27T03:07:11.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.178+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70940038a0 con 0x7f70a00af900 2024-03-27T03:07:11.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.178+0000 7f70877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:11.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.178+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f7094003bb0 con 0x7f70a00af900 2024-03-27T03:07:11.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.178+0000 7f70877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.178+0000 7f70877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.178+0000 7f70877fe640 10 monclient: dump: 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.178+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f7094019b80 con 0x7f70a00af900 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.178+0000 7f709cff9640 4 set_mon_vals no callback set 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.179+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f7094083870 con 0x7f70a00af900 2024-03-27T03:07:11.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.179+0000 7f70877fe640 10 client.4281.objecter ms_dispatch 0x7f70a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:07:11.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.179+0000 7f70877fe640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:07:11.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.179+0000 7f70877fe640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:07:11.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.180+0000 7f709effd640 10 monclient: get_auth_request con 0x7f708c06e040 auth_method 0 2024-03-27T03:07:11.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.180+0000 7f70877fe640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-27T03:07:11.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.180+0000 7f70877fe640 10 client.4281.objecter ms_handle_connect 0x7f708c06e040 2024-03-27T03:07:11.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.187+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f709404f2c0 con 0x7f70a00af900 2024-03-27T03:07:11.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.187+0000 7f70877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:11.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.187+0000 7f70877fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:07:11.506 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.513+0000 7f70a55d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:11.506 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.513+0000 7f70a55d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:11.506 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.513+0000 7f70a55d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:07:11.506 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.513+0000 7f70a55d6640 1 -- 172.21.15.153:0/1427557835 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70a01e4d00 con 0x7f70a00af900 2024-03-27T03:07:11.507 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.514+0000 7f70877fe640 1 -- 172.21.15.153:0/1427557835 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f70940511f0 con 0x7f70a00af900 2024-03-27T03:07:11.507 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.514+0000 7f70877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:11.507 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.514+0000 7f70877fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:07:11.507 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:07:11.507 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:07:11.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.517+0000 7f70a55d6640 10 librados: watch_flush enter 2024-03-27T03:07:11.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.517+0000 7f70a55d6640 10 librados: watch_flush exit 2024-03-27T03:07:11.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.517+0000 7f70a55d6640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-27T03:07:11.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.517+0000 7f70a55d6640 10 client.4281.objecter successfully canceled tick 2024-03-27T03:07:11.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.518+0000 7f70a55d6640 1 -- 172.21.15.153:0/1427557835 >> v1:172.21.15.153:6816/2637937781 conn(0x7f708c06e040 legacy=0x7f708c070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:11.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.518+0000 7f70a55d6640 10 monclient: shutdown 2024-03-27T03:07:11.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.518+0000 7f70a55d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:11.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.518+0000 7f70a55d6640 1 -- 172.21.15.153:0/1427557835 >> v1:172.21.15.153:6789/0 conn(0x7f70a00af900 legacy=0x7f70a01e42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:11.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.518+0000 7f70a55d6640 1 -- 172.21.15.153:0/1427557835 shutdown_connections 2024-03-27T03:07:11.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.518+0000 7f70a55d6640 1 -- 172.21.15.153:0/1427557835 >> 172.21.15.153:0/1427557835 conn(0x7f70a00b35e0 msgr2=0x7f70a00adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:11.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.519+0000 7f70a55d6640 1 -- 172.21.15.153:0/1427557835 shutdown_connections 2024-03-27T03:07:11.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.519+0000 7f70a55d6640 1 -- 172.21.15.153:0/1427557835 wait complete. 2024-03-27T03:07:11.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:11.519+0000 7f70a55d6640 1 librados: shutdown 2024-03-27T03:07:11.530 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:07:18.337 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:07:18.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.493+0000 7eff9c84a640 10 monclient: get_monmap_and_config 2024-03-27T03:07:18.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.493+0000 7eff9c84a640 10 monclient: build_initial_monmap 2024-03-27T03:07:18.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.493+0000 7eff9c84a640 10 monclient: monmap: 2024-03-27T03:07:18.487 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:18.487 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:18.487 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:07:18.494396+0000 2024-03-27T03:07:18.487 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:07:18.494396+0000 2024-03-27T03:07:18.487 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:18.487 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:18.487 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:18.487 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:07:18.487 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:07:18.487 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:18.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.496+0000 7eff9c84a640 1 Processor -- start 2024-03-27T03:07:18.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.496+0000 7eff9c84a640 1 -- start start 2024-03-27T03:07:18.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.496+0000 7eff9c84a640 10 monclient: init 2024-03-27T03:07:18.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.497+0000 7eff9c84a640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:18.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.497+0000 7eff9c84a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:07:18.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.497+0000 7eff9c84a640 10 monclient(hunting): picked mon.noname-a con 0x7eff980b4790 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:18.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.498+0000 7eff9c84a640 10 monclient(hunting): picked mon.noname-b con 0x7eff980aea90 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:18.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.498+0000 7eff9c84a640 10 monclient(hunting): picked mon.noname-c con 0x7eff980b39f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:18.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.498+0000 7eff9c84a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff980b0c60 con 0x7eff980b4790 2024-03-27T03:07:18.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.498+0000 7eff9c84a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff980af020 con 0x7eff980b39f0 2024-03-27T03:07:18.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.498+0000 7eff9c84a640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff9813fbd0 con 0x7eff980aea90 2024-03-27T03:07:18.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.498+0000 7eff9c84a640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:18.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.498+0000 7eff9c84a640 10 monclient(hunting): authenticate will time out at 1398.803974s 2024-03-27T03:07:18.494 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.501+0000 7eff95d9b640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7eff980aea90 0x7eff980af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:56777/0 (socket says 172.21.15.153:56777) 2024-03-27T03:07:18.494 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.501+0000 7eff95d9b640 1 -- 172.21.15.153:0/1632407263 learned_addr learned my addr 172.21.15.153:0/1632407263 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:18.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.505+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023344875 0 0) 0x7eff980af020 con 0x7eff980b39f0 2024-03-27T03:07:18.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.505+0000 7eff9559a640 10 monclient(hunting): _negotiate 2024-03-27T03:07:18.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.505+0000 7eff9559a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:18.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.505+0000 7eff9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:18.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.505+0000 7eff9559a640 10 monclient(hunting): my global_id is 4291 2024-03-27T03:07:18.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.505+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff80003520 con 0x7eff980b39f0 2024-03-27T03:07:18.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.505+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514834193 0 0) 0x7eff9813fbd0 con 0x7eff980aea90 2024-03-27T03:07:18.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.505+0000 7eff9559a640 10 monclient(hunting): _negotiate 2024-03-27T03:07:18.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.505+0000 7eff9559a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:18.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.505+0000 7eff9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:18.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.505+0000 7eff9559a640 10 monclient(hunting): my global_id is 4304 2024-03-27T03:07:18.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.505+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff980af020 con 0x7eff980aea90 2024-03-27T03:07:18.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196306382 0 0) 0x7eff980b0c60 con 0x7eff980b4790 2024-03-27T03:07:18.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 10 monclient(hunting): _negotiate 2024-03-27T03:07:18.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:18.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:18.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 10 monclient(hunting): my global_id is 4284 2024-03-27T03:07:18.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff9813fbd0 con 0x7eff980b4790 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3570905410 0 0) 0x7eff80003520 con 0x7eff980b39f0 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 >> v1:172.21.15.153:6790/0 conn(0x7eff980aea90 legacy=0x7eff980af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 >> v1:172.21.15.153:6789/0 conn(0x7eff980b4790 legacy=0x7eff980b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 10 monclient: _finish_hunting 0 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 1 monclient: found mon.noname-c 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9813fd60 con 0x7eff980b39f0 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 10 monclient: _finish_auth 0 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 10 monclient: _check_auth_tickets 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9c84a640 5 monclient: authenticate success, global_id 4291 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7eff8c001690 con 0x7eff980b39f0 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:07:18.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9559a640 10 monclient: dump: 2024-03-27T03:07:18.501 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:18.501 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:18.501 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:18.501 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:18.501 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:18.501 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:18.501 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:18.501 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:18.501 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:18.501 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:18.501 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.506+0000 7eff9c84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:07:18.502 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.509+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff8c001870 con 0x7eff980b39f0 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.509+0000 7eff9559a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.509+0000 7eff9559a640 1 -- 172.21.15.153:0/1632407263 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7eff8c001690 con 0x7eff980b39f0 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.509+0000 7eff9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.509+0000 7eff9559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.509+0000 7eff9559a640 10 monclient: dump: 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:18.503 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:18.504 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.509+0000 7eff9c84a640 10 monclient: get_monmap_and_config success 2024-03-27T03:07:18.504 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.509+0000 7eff9c84a640 4 set_mon_vals no callback set 2024-03-27T03:07:18.504 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.510+0000 7eff9c84a640 10 monclient: shutdown 2024-03-27T03:07:18.504 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.511+0000 7eff9c84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:18.504 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.511+0000 7eff9c84a640 1 -- 172.21.15.153:0/1632407263 >> v1:172.21.15.196:6789/0 conn(0x7eff980b39f0 legacy=0x7eff9813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:18.505 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.512+0000 7eff9c84a640 1 -- 172.21.15.153:0/1632407263 shutdown_connections 2024-03-27T03:07:18.505 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.512+0000 7eff9c84a640 1 -- 172.21.15.153:0/1632407263 >> 172.21.15.153:0/1632407263 conn(0x7eff98066a00 msgr2=0x7eff98066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:18.508 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.515+0000 7eff9c84a640 1 -- 172.21.15.153:0/1632407263 shutdown_connections 2024-03-27T03:07:18.510 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.517+0000 7eff9c84a640 1 -- 172.21.15.153:0/1632407263 wait complete. 2024-03-27T03:07:18.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.518+0000 7eff9c84a640 10 monclient: build_initial_monmap 2024-03-27T03:07:18.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.518+0000 7eff9c84a640 10 monclient: monmap: 2024-03-27T03:07:18.511 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:18.511 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:07:18.511 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:07:18.511 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:07:18.511 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:18.512 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:18.512 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:18.512 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:07:18.512 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:07:18.512 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:18.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.519+0000 7eff9c84a640 1 librados: starting msgr at 2024-03-27T03:07:18.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.519+0000 7eff9c84a640 1 librados: starting objecter 2024-03-27T03:07:18.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.519+0000 7eff9c84a640 1 Processor -- start 2024-03-27T03:07:18.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.521+0000 7eff9c84a640 1 -- start start 2024-03-27T03:07:18.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.521+0000 7eff9c84a640 1 librados: setting wanted keys 2024-03-27T03:07:18.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.521+0000 7eff9c84a640 1 librados: calling monclient init 2024-03-27T03:07:18.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.521+0000 7eff9c84a640 10 monclient: init 2024-03-27T03:07:18.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.522+0000 7eff9c84a640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:18.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.522+0000 7eff9c84a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:07:18.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.522+0000 7eff9c84a640 10 monclient(hunting): picked mon.noname-b con 0x7eff980b39f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:18.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.522+0000 7eff9c84a640 10 monclient(hunting): picked mon.noname-a con 0x7eff980b4790 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:18.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.522+0000 7eff9c84a640 10 monclient(hunting): picked mon.noname-c con 0x7eff980a9ad0 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:18.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.522+0000 7eff9c84a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff981d4de0 con 0x7eff980b4790 2024-03-27T03:07:18.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.522+0000 7eff9c84a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff981d7520 con 0x7eff980b39f0 2024-03-27T03:07:18.519 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.522+0000 7eff9c84a640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff980aaef0 con 0x7eff980a9ad0 2024-03-27T03:07:18.519 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.522+0000 7eff9c84a640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:18.519 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.522+0000 7eff9c84a640 10 monclient(hunting): authenticate will time out at 1398.828030s 2024-03-27T03:07:18.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.528+0000 7eff96d9d640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7eff980a9ad0 0x7eff981e1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:57021/0 (socket says 172.21.15.153:57021) 2024-03-27T03:07:18.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.528+0000 7eff96d9d640 1 -- 172.21.15.153:0/3438054324 learned_addr learned my addr 172.21.15.153:0/3438054324 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:18.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.528+0000 7eff86ffd640 10 client.?.objecter ms_handle_connect 0x7eff980b39f0 2024-03-27T03:07:18.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.528+0000 7eff86ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:18.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 10 client.?.objecter ms_handle_connect 0x7eff980a9ad0 2024-03-27T03:07:18.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:18.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051602298 0 0) 0x7eff981d7520 con 0x7eff980b39f0 2024-03-27T03:07:18.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:07:18.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:18.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:18.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 10 monclient(hunting): my global_id is 4294 2024-03-27T03:07:18.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff78004eb0 con 0x7eff980b39f0 2024-03-27T03:07:18.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 10 client.?.objecter ms_handle_connect 0x7eff980b4790 2024-03-27T03:07:18.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 28985115 0 0) 0x7eff78004eb0 con 0x7eff980b39f0 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff981d7520 con 0x7eff980b39f0 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261473924 0 0) 0x7eff980aaef0 con 0x7eff980a9ad0 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.529+0000 7eff86ffd640 10 monclient(hunting): my global_id is 4307 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff78005590 con 0x7eff980a9ad0 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7eff90001bd0 con 0x7eff980b39f0 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 10 monclient(hunting): dump: 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:18.523 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113044015 0 0) 0x7eff981d4de0 con 0x7eff980b4790 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 10 monclient(hunting): my global_id is 4287 2024-03-27T03:07:18.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff78005a10 con 0x7eff980b4790 2024-03-27T03:07:18.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2503648551 0 0) 0x7eff78005590 con 0x7eff980a9ad0 2024-03-27T03:07:18.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff981d4de0 con 0x7eff980a9ad0 2024-03-27T03:07:18.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7eff8c001690 con 0x7eff980a9ad0 2024-03-27T03:07:18.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:18.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:07:18.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 10 monclient(hunting): dump: 2024-03-27T03:07:18.525 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:18.525 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:18.525 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:18.526 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:18.526 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:18.526 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:18.526 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:18.526 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:18.526 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:18.526 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:18.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1026809418 0 0) 0x7eff78005a10 con 0x7eff980b4790 2024-03-27T03:07:18.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.530+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff78006380 con 0x7eff980b4790 2024-03-27T03:07:18.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7eff88001930 con 0x7eff980b4790 2024-03-27T03:07:18.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:18.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:07:18.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 10 monclient(hunting): dump: 2024-03-27T03:07:18.527 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:18.527 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:18.527 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:18.527 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:18.527 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:18.527 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:18.527 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:18.527 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4193913753 0 0) 0x7eff981d4de0 con 0x7eff980a9ad0 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 >> v1:172.21.15.196:6789/0 conn(0x7eff980b39f0 legacy=0x7eff981d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 >> v1:172.21.15.153:6789/0 conn(0x7eff980b4790 legacy=0x7eff980aa7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 1 monclient: found mon.c 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff981d71f0 con 0x7eff980a9ad0 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 10 monclient: _finish_auth 0 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff9c84a640 5 monclient: authenticate success, global_id 4307 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff9c84a640 10 monclient: _renew_subs 2024-03-27T03:07:18.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff9c84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:18.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff9c84a640 1 -- 172.21.15.153:0/3438054324 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff981e3140 con 0x7eff980a9ad0 2024-03-27T03:07:18.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff9c84a640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:07:18.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff9c84a640 10 monclient: _renew_subs 2024-03-27T03:07:18.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff9c84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:18.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff9c84a640 1 -- 172.21.15.153:0/3438054324 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff981e34b0 con 0x7eff980a9ad0 2024-03-27T03:07:18.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.531+0000 7eff9c84a640 1 librados: init done 2024-03-27T03:07:18.530 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.535+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff8c001020 con 0x7eff980a9ad0 2024-03-27T03:07:18.530 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.535+0000 7eff86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:18.530 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.535+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7eff8c001690 con 0x7eff980a9ad0 2024-03-27T03:07:18.530 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.535+0000 7eff86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:18.530 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.535+0000 7eff86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:07:18.530 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.535+0000 7eff86ffd640 10 monclient: dump: 2024-03-27T03:07:18.530 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:18.530 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:18.530 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:18.530 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:18.530 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:18.531 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:18.531 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:18.531 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:18.531 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:18.531 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:18.531 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.535+0000 7eff94d99640 4 set_mon_vals no callback set 2024-03-27T03:07:18.531 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.536+0000 7eff9c84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:18.531 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.536+0000 7eff9c84a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:18.531 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.536+0000 7eff9c84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:18.531 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.536+0000 7eff9c84a640 1 -- 172.21.15.153:0/3438054324 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7c005080 con 0x7eff980a9ad0 2024-03-27T03:07:18.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.537+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7eff8c003a70 con 0x7eff980a9ad0 2024-03-27T03:07:18.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.538+0000 7eff9659c640 10 monclient: get_auth_request con 0x7eff7806e940 auth_method 0 2024-03-27T03:07:18.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.539+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7eff8c0544d0 con 0x7eff980a9ad0 2024-03-27T03:07:18.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.539+0000 7eff86ffd640 10 client.4307.objecter ms_dispatch 0x7eff98000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:07:18.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.539+0000 7eff86ffd640 3 client.4307.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:07:18.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.539+0000 7eff86ffd640 3 client.4307.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:07:18.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.539+0000 7eff86ffd640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-27T03:07:18.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.539+0000 7eff86ffd640 10 client.4307.objecter ms_handle_connect 0x7eff7806e940 2024-03-27T03:07:18.542 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.550+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff8c050770 con 0x7eff980a9ad0 2024-03-27T03:07:18.542 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.550+0000 7eff86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:18.542 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.550+0000 7eff86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:07:18.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.886+0000 7eff9c84a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:18.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.886+0000 7eff9c84a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:18.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.886+0000 7eff9c84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:18.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.886+0000 7eff9c84a640 1 -- 172.21.15.153:0/3438054324 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff7c005330 con 0x7eff980a9ad0 2024-03-27T03:07:18.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.890+0000 7eff86ffd640 1 -- 172.21.15.153:0/3438054324 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7eff8c050770 con 0x7eff980a9ad0 2024-03-27T03:07:18.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.890+0000 7eff86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:18.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.890+0000 7eff86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:07:18.885 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:07:18.885 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:07:18.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.896+0000 7eff9c84a640 10 librados: watch_flush enter 2024-03-27T03:07:18.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.896+0000 7eff9c84a640 10 librados: watch_flush exit 2024-03-27T03:07:18.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.896+0000 7eff9c84a640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-27T03:07:18.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.896+0000 7eff9c84a640 10 client.4307.objecter successfully canceled tick 2024-03-27T03:07:18.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.896+0000 7eff9c84a640 1 -- 172.21.15.153:0/3438054324 >> v1:172.21.15.153:6816/2637937781 conn(0x7eff7806e940 legacy=0x7eff78070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:18.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.896+0000 7eff9c84a640 10 monclient: shutdown 2024-03-27T03:07:18.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.896+0000 7eff9c84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:18.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.896+0000 7eff9c84a640 1 -- 172.21.15.153:0/3438054324 >> v1:172.21.15.153:6790/0 conn(0x7eff980a9ad0 legacy=0x7eff981e1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:18.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.897+0000 7eff9c84a640 1 -- 172.21.15.153:0/3438054324 shutdown_connections 2024-03-27T03:07:18.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.897+0000 7eff9c84a640 1 -- 172.21.15.153:0/3438054324 >> 172.21.15.153:0/3438054324 conn(0x7eff98066a00 msgr2=0x7eff980b1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:18.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.897+0000 7eff9c84a640 1 -- 172.21.15.153:0/3438054324 shutdown_connections 2024-03-27T03:07:18.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.897+0000 7eff9c84a640 1 -- 172.21.15.153:0/3438054324 wait complete. 2024-03-27T03:07:18.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:18.897+0000 7eff9c84a640 1 librados: shutdown 2024-03-27T03:07:18.907 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:07:25.713 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:07:25.863 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.869+0000 7f85230a2640 10 monclient: get_monmap_and_config 2024-03-27T03:07:25.863 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.870+0000 7f85230a2640 10 monclient: build_initial_monmap 2024-03-27T03:07:25.863 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.870+0000 7f85230a2640 10 monclient: monmap: 2024-03-27T03:07:25.863 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:25.863 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:25.863 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:07:25.870948+0000 2024-03-27T03:07:25.863 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:07:25.870948+0000 2024-03-27T03:07:25.863 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:25.863 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:25.863 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:25.863 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:07:25.863 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:07:25.864 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:25.865 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.871+0000 7f85230a2640 1 Processor -- start 2024-03-27T03:07:25.865 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.871+0000 7f85230a2640 1 -- start start 2024-03-27T03:07:25.865 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.871+0000 7f85230a2640 10 monclient: init 2024-03-27T03:07:25.866 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.872+0000 7f85230a2640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:25.866 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.872+0000 7f85230a2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:07:25.866 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.872+0000 7f85230a2640 10 monclient(hunting): picked mon.noname-a con 0x7f8514006a70 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:25.868 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.875+0000 7f85230a2640 10 monclient(hunting): picked mon.noname-c con 0x7f851402b100 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:25.868 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.875+0000 7f852189f640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f851402b100 0x7f85140cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:37225/0 (socket says 172.21.15.153:37225) 2024-03-27T03:07:25.868 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.876+0000 7f852189f640 1 -- 172.21.15.153:0/2633695194 learned_addr learned my addr 172.21.15.153:0/2633695194 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:25.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.878+0000 7f85230a2640 10 monclient(hunting): picked mon.noname-b con 0x7f8514029c30 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:25.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.878+0000 7f85230a2640 1 -- 172.21.15.153:0/2633695194 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851402a270 con 0x7f8514006a70 2024-03-27T03:07:25.871 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.878+0000 7f85230a2640 1 -- 172.21.15.153:0/2633695194 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851402c440 con 0x7f851402b100 2024-03-27T03:07:25.871 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.878+0000 7f85230a2640 1 -- 172.21.15.153:0/2633695194 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85140c9440 con 0x7f8514029c30 2024-03-27T03:07:25.871 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.878+0000 7f85230a2640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:25.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.878+0000 7f85230a2640 10 monclient(hunting): authenticate will time out at 1406.183804s 2024-03-27T03:07:25.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.878+0000 7f852109e640 1 -- 172.21.15.153:0/2633695194 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052893387 0 0) 0x7f851402c440 con 0x7f851402b100 2024-03-27T03:07:25.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.878+0000 7f852109e640 10 monclient(hunting): _negotiate 2024-03-27T03:07:25.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.878+0000 7f852109e640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:25.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.878+0000 7f852109e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:25.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.878+0000 7f852109e640 10 monclient(hunting): my global_id is 4297 2024-03-27T03:07:25.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.878+0000 7f852109e640 1 -- 172.21.15.153:0/2633695194 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8508003520 con 0x7f851402b100 2024-03-27T03:07:25.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 1 -- 172.21.15.153:0/2633695194 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3486116250 0 0) 0x7f8508003520 con 0x7f851402b100 2024-03-27T03:07:25.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 1 -- 172.21.15.153:0/2633695194 >> v1:172.21.15.153:6790/0 conn(0x7f8514029c30 legacy=0x7f85140d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:07:25.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 1 -- 172.21.15.153:0/2633695194 >> v1:172.21.15.153:6789/0 conn(0x7f8514006a70 legacy=0x7f85140c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:07:25.878 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 10 monclient: _finish_hunting 0 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 1 monclient: found mon.noname-c 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 1 -- 172.21.15.153:0/2633695194 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85140c95d0 con 0x7f851402b100 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 10 monclient: _finish_auth 0 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 10 monclient: _check_auth_tickets 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 1 -- 172.21.15.153:0/2633695194 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8518001690 con 0x7f851402b100 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 10 monclient: dump: 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:25.879 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f85230a2640 5 monclient: authenticate success, global_id 4297 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f85230a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 1 -- 172.21.15.153:0/2633695194 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8518001860 con 0x7f851402b100 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 1 -- 172.21.15.153:0/2633695194 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8518001cb0 con 0x7f851402b100 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f852109e640 10 monclient: dump: 2024-03-27T03:07:25.880 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f85230a2640 10 monclient: get_monmap_and_config success 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.879+0000 7f85230a2640 4 set_mon_vals no callback set 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.880+0000 7f85230a2640 10 monclient: shutdown 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.881+0000 7f85230a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:25.881 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.881+0000 7f85230a2640 1 -- 172.21.15.153:0/2633695194 >> v1:172.21.15.196:6789/0 conn(0x7f851402b100 legacy=0x7f85140cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.881+0000 7f85230a2640 1 -- 172.21.15.153:0/2633695194 shutdown_connections 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.881+0000 7f85230a2640 1 -- 172.21.15.153:0/2633695194 >> 172.21.15.153:0/2633695194 conn(0x7f85140058f0 msgr2=0x7f8514005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.881+0000 7f85230a2640 1 -- 172.21.15.153:0/2633695194 shutdown_connections 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.882+0000 7f85230a2640 1 -- 172.21.15.153:0/2633695194 wait complete. 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.883+0000 7f85230a2640 10 monclient: build_initial_monmap 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.883+0000 7f85230a2640 10 monclient: monmap: 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:07:25.882 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:07:25.883 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:25.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.883+0000 7f85230a2640 1 librados: starting msgr at 2024-03-27T03:07:25.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.883+0000 7f85230a2640 1 librados: starting objecter 2024-03-27T03:07:25.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.883+0000 7f85230a2640 1 Processor -- start 2024-03-27T03:07:25.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 1 -- start start 2024-03-27T03:07:25.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 1 librados: setting wanted keys 2024-03-27T03:07:25.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 1 librados: calling monclient init 2024-03-27T03:07:25.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 10 monclient: init 2024-03-27T03:07:25.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:25.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:07:25.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 10 monclient(hunting): picked mon.noname-c con 0x7f8514006a70 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 10 monclient(hunting): picked mon.noname-a con 0x7f8514029c30 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 10 monclient(hunting): picked mon.noname-b con 0x7f8514032460 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85140c9a10 con 0x7f8514029c30 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851415e520 con 0x7f8514032460 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85140cb640 con 0x7f8514006a70 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.884+0000 7f85230a2640 10 monclient(hunting): authenticate will time out at 1406.190479s 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.889+0000 7f85220a0640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f8514006a70 0x7f85140ca370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:52567/0 (socket says 172.21.15.153:52567) 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.889+0000 7f85220a0640 1 -- 172.21.15.153:0/722935966 learned_addr learned my addr 172.21.15.153:0/722935966 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.891+0000 7f8512ffd640 10 client.?.objecter ms_handle_connect 0x7f8514006a70 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.891+0000 7f8512ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.892+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620232711 0 0) 0x7f85140cb640 con 0x7f8514006a70 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.892+0000 7f8512ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:07:25.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.892+0000 7f8512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:25.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.892+0000 7f8512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:25.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.892+0000 7f8512ffd640 10 monclient(hunting): my global_id is 4310 2024-03-27T03:07:25.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.892+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8500004eb0 con 0x7f8514006a70 2024-03-27T03:07:25.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.892+0000 7f8512ffd640 10 client.?.objecter ms_handle_connect 0x7f8514032460 2024-03-27T03:07:25.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.892+0000 7f8512ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:25.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.895+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567347454 0 0) 0x7f851415e520 con 0x7f8514032460 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.895+0000 7f8512ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.895+0000 7f8512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.895+0000 7f8512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.895+0000 7f8512ffd640 10 monclient(hunting): my global_id is 4300 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85140cb640 con 0x7f8514032460 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 10 client.?.objecter ms_handle_connect 0x7f8514029c30 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1619000774 0 0) 0x7f8500004eb0 con 0x7f8514006a70 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f851415e520 con 0x7f8514006a70 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227629489 0 0) 0x7f85140c9a10 con 0x7f8514029c30 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:25.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 10 monclient(hunting): my global_id is 4290 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8500004eb0 con 0x7f8514029c30 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f850c001960 con 0x7f8514006a70 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 10 monclient(hunting): dump: 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:25.894 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4052099243 0 0) 0x7f851415e520 con 0x7f8514006a70 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 >> v1:172.21.15.196:6789/0 conn(0x7f8514032460 legacy=0x7f8514160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.896+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 >> v1:172.21.15.153:6789/0 conn(0x7f8514029c30 legacy=0x7f8514031d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f8512ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f8512ffd640 1 monclient: found mon.c 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f8512ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f8512ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85140cb080 con 0x7f8514006a70 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f8512ffd640 10 monclient: _finish_auth 0 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f8512ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f8512ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f85230a2640 5 monclient: authenticate success, global_id 4310 2024-03-27T03:07:25.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f85230a2640 10 monclient: _renew_subs 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f85230a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f85230a2640 1 -- 172.21.15.153:0/722935966 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85140cb2d0 con 0x7f8514006a70 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f85230a2640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f85230a2640 10 monclient: _renew_subs 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f85230a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f85230a2640 1 -- 172.21.15.153:0/722935966 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85141615b0 con 0x7f8514006a70 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.897+0000 7f85230a2640 1 librados: init done 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.899+0000 7f8510ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.899+0000 7f8510ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.899+0000 7f8510ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.899+0000 7f8510ff9640 1 -- 172.21.15.153:0/722935966 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8514004cd0 con 0x7f8514006a70 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.899+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f850c001d00 con 0x7f8514006a70 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.899+0000 7f8512ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.899+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f850c001b80 con 0x7f8514006a70 2024-03-27T03:07:25.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.899+0000 7f8512ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.900+0000 7f8512ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.900+0000 7f8512ffd640 10 monclient: dump: 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:25.897 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.899+0000 7f852089d640 4 set_mon_vals no callback set 2024-03-27T03:07:25.898 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.901+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f850c003ac0 con 0x7f8514006a70 2024-03-27T03:07:25.898 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.902+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f850c084af0 con 0x7f8514006a70 2024-03-27T03:07:25.898 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.902+0000 7f8512ffd640 10 client.4310.objecter ms_dispatch 0x7f851c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:07:25.898 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.902+0000 7f8512ffd640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:07:25.898 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.902+0000 7f8512ffd640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:07:25.898 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.902+0000 7f8512ffd640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-27T03:07:25.898 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.906+0000 7f852189f640 10 monclient: get_auth_request con 0x7f850006dad0 auth_method 0 2024-03-27T03:07:25.899 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.906+0000 7f8512ffd640 10 client.4310.objecter ms_handle_connect 0x7f850006dad0 2024-03-27T03:07:25.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.910+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f850c0503d0 con 0x7f8514006a70 2024-03-27T03:07:25.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.910+0000 7f8512ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:25.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:25.910+0000 7f8512ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:07:26.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.247+0000 7f8510ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:26.241 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.247+0000 7f8510ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:26.241 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.247+0000 7f8510ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:26.241 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.247+0000 7f8510ff9640 1 -- 172.21.15.153:0/722935966 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85140008d0 con 0x7f8514006a70 2024-03-27T03:07:26.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.250+0000 7f8512ffd640 1 -- 172.21.15.153:0/722935966 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f850c052300 con 0x7f8514006a70 2024-03-27T03:07:26.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.250+0000 7f8512ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:26.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.250+0000 7f8512ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:07:26.244 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:07:26.244 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:07:26.247 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.254+0000 7f8510ff9640 10 librados: watch_flush enter 2024-03-27T03:07:26.247 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.254+0000 7f8510ff9640 10 librados: watch_flush exit 2024-03-27T03:07:26.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.254+0000 7f8510ff9640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-27T03:07:26.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.254+0000 7f8510ff9640 10 client.4310.objecter successfully canceled tick 2024-03-27T03:07:26.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.255+0000 7f8510ff9640 1 -- 172.21.15.153:0/722935966 >> v1:172.21.15.153:6816/2637937781 conn(0x7f850006dad0 legacy=0x7f850006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:26.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.255+0000 7f8510ff9640 10 monclient: shutdown 2024-03-27T03:07:26.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.255+0000 7f8510ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:26.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.255+0000 7f8510ff9640 1 -- 172.21.15.153:0/722935966 >> v1:172.21.15.153:6790/0 conn(0x7f8514006a70 legacy=0x7f85140ca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:26.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.255+0000 7f8510ff9640 1 -- 172.21.15.153:0/722935966 shutdown_connections 2024-03-27T03:07:26.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.255+0000 7f8510ff9640 1 -- 172.21.15.153:0/722935966 >> 172.21.15.153:0/722935966 conn(0x7f85140058f0 msgr2=0x7f8514032d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:26.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.255+0000 7f8510ff9640 1 -- 172.21.15.153:0/722935966 shutdown_connections 2024-03-27T03:07:26.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.255+0000 7f8510ff9640 1 -- 172.21.15.153:0/722935966 wait complete. 2024-03-27T03:07:26.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:26.255+0000 7f8510ff9640 1 librados: shutdown 2024-03-27T03:07:26.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:07:33.072 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:07:33.224 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.231+0000 7f758ab5d640 10 monclient: get_monmap_and_config 2024-03-27T03:07:33.225 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.231+0000 7f758ab5d640 10 monclient: build_initial_monmap 2024-03-27T03:07:33.225 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.231+0000 7f758ab5d640 10 monclient: monmap: 2024-03-27T03:07:33.225 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:33.225 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:33.225 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:07:33.232568+0000 2024-03-27T03:07:33.225 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:07:33.232568+0000 2024-03-27T03:07:33.225 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:33.225 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:33.225 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:33.225 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:07:33.225 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:07:33.225 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:33.226 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.232+0000 7f758ab5d640 1 Processor -- start 2024-03-27T03:07:33.226 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.232+0000 7f758ab5d640 1 -- start start 2024-03-27T03:07:33.226 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.232+0000 7f758ab5d640 10 monclient: init 2024-03-27T03:07:33.226 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.233+0000 7f758ab5d640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:33.226 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.233+0000 7f758ab5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:07:33.226 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.233+0000 7f758ab5d640 10 monclient(hunting): picked mon.noname-b con 0x7f75840b0860 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:33.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.233+0000 7f758ab5d640 10 monclient(hunting): picked mon.noname-a con 0x7f758413ef30 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:33.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.233+0000 7f758ab5d640 10 monclient(hunting): picked mon.noname-c con 0x7f75840af8e0 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:33.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.233+0000 7f758ab5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840b0c40 con 0x7f758413ef30 2024-03-27T03:07:33.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.233+0000 7f758ab5d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840aed10 con 0x7f75840af8e0 2024-03-27T03:07:33.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.233+0000 7f758ab5d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840afd50 con 0x7f75840b0860 2024-03-27T03:07:33.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.233+0000 7f758ab5d640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:33.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.233+0000 7f758ab5d640 10 monclient(hunting): authenticate will time out at 1413.539256s 2024-03-27T03:07:33.230 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.238+0000 7f75888f9640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f75840b0860 0x7f758413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:45139/0 (socket says 172.21.15.153:45139) 2024-03-27T03:07:33.230 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.238+0000 7f75888f9640 1 -- 172.21.15.153:0/1791716381 learned_addr learned my addr 172.21.15.153:0/1791716381 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.239+0000 7f75837fe640 1 -- 172.21.15.153:0/1791716381 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174076210 0 0) 0x7f75840afd50 con 0x7f75840b0860 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.239+0000 7f75837fe640 10 monclient(hunting): _negotiate 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.239+0000 7f75837fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.239+0000 7f75837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.239+0000 7f75837fe640 10 monclient(hunting): my global_id is 4313 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.239+0000 7f75837fe640 1 -- 172.21.15.153:0/1791716381 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f756c003520 con 0x7f75840b0860 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.240+0000 7f75837fe640 1 -- 172.21.15.153:0/1791716381 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 855122233 0 0) 0x7f756c003520 con 0x7f75840b0860 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.240+0000 7f75837fe640 1 -- 172.21.15.153:0/1791716381 >> v1:172.21.15.196:6789/0 conn(0x7f75840af8e0 legacy=0x7f7584149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.240+0000 7f75837fe640 1 -- 172.21.15.153:0/1791716381 >> v1:172.21.15.153:6789/0 conn(0x7f758413ef30 legacy=0x7f75840af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.240+0000 7f75837fe640 10 monclient: _finish_hunting 0 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.240+0000 7f75837fe640 1 monclient: found mon.noname-b 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.240+0000 7f75837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.240+0000 7f75837fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:07:33.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.240+0000 7f75837fe640 1 -- 172.21.15.153:0/1791716381 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75840ab1b0 con 0x7f75840b0860 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.240+0000 7f75837fe640 10 monclient: _finish_auth 0 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.240+0000 7f75837fe640 10 monclient: _check_auth_tickets 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.240+0000 7f75837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.240+0000 7f75837fe640 1 -- 172.21.15.153:0/1791716381 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f7574001690 con 0x7f75840b0860 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.241+0000 7f758ab5d640 5 monclient: authenticate success, global_id 4313 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.241+0000 7f758ab5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.241+0000 7f75837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.241+0000 7f75837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.241+0000 7f75837fe640 10 monclient: dump: 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:33.236 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.241+0000 7f75837fe640 1 -- 172.21.15.153:0/1791716381 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7574001920 con 0x7f75840b0860 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.241+0000 7f758ab5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.241+0000 7f75837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.241+0000 7f75837fe640 1 -- 172.21.15.153:0/1791716381 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f7574001cb0 con 0x7f75840b0860 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.241+0000 7f758ab5d640 10 monclient: get_monmap_and_config success 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.241+0000 7f758ab5d640 4 set_mon_vals no callback set 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.242+0000 7f75837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.242+0000 7f75837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.242+0000 7f75837fe640 10 monclient: dump: 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:33.237 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:33.238 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:33.238 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:33.238 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:33.238 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:33.241 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.242+0000 7f758ab5d640 10 monclient: shutdown 2024-03-27T03:07:33.241 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.242+0000 7f758ab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.242+0000 7f758ab5d640 1 -- 172.21.15.153:0/1791716381 >> v1:172.21.15.153:6790/0 conn(0x7f75840b0860 legacy=0x7f758413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.242+0000 7f758ab5d640 1 -- 172.21.15.153:0/1791716381 shutdown_connections 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.242+0000 7f758ab5d640 1 -- 172.21.15.153:0/1791716381 >> 172.21.15.153:0/1791716381 conn(0x7f758406b660 msgr2=0x7f758406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.243+0000 7f758ab5d640 1 -- 172.21.15.153:0/1791716381 shutdown_connections 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.243+0000 7f758ab5d640 1 -- 172.21.15.153:0/1791716381 wait complete. 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.244+0000 7f758ab5d640 10 monclient: build_initial_monmap 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.244+0000 7f758ab5d640 10 monclient: monmap: 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:33.242 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.244+0000 7f758ab5d640 1 librados: starting msgr at 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.244+0000 7f758ab5d640 1 librados: starting objecter 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.244+0000 7f758ab5d640 1 Processor -- start 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.245+0000 7f758ab5d640 1 -- start start 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.245+0000 7f758ab5d640 1 librados: setting wanted keys 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.245+0000 7f758ab5d640 1 librados: calling monclient init 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.245+0000 7f758ab5d640 10 monclient: init 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.246+0000 7f758ab5d640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.246+0000 7f758ab5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:07:33.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.246+0000 7f758ab5d640 10 monclient(hunting): picked mon.noname-a con 0x7f75840af8e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:33.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.249+0000 7f758ab5d640 10 monclient(hunting): picked mon.noname-c con 0x7f75840b0860 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:33.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.250+0000 7f758ab5d640 10 monclient(hunting): picked mon.noname-b con 0x7f758413ef30 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:33.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.250+0000 7f758ab5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840b17b0 con 0x7f75840af8e0 2024-03-27T03:07:33.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.250+0000 7f758ab5d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758413b050 con 0x7f758413ef30 2024-03-27T03:07:33.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.250+0000 7f758ab5d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758413a740 con 0x7f75840b0860 2024-03-27T03:07:33.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.250+0000 7f758ab5d640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:33.245 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.250+0000 7f758ab5d640 10 monclient(hunting): authenticate will time out at 1413.556262s 2024-03-27T03:07:33.247 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75890fa640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f758413ef30 0x7f75841d59c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:60981/0 (socket says 172.21.15.153:60981) 2024-03-27T03:07:33.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75890fa640 1 -- 172.21.15.153:0/1441090443 learned_addr learned my addr 172.21.15.153:0/1441090443 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:33.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75817fa640 10 client.?.objecter ms_handle_connect 0x7f758413ef30 2024-03-27T03:07:33.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75817fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:33.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75817fa640 10 client.?.objecter ms_handle_connect 0x7f75840b0860 2024-03-27T03:07:33.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75817fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:33.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75817fa640 10 client.?.objecter ms_handle_connect 0x7f75840af8e0 2024-03-27T03:07:33.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75817fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:33.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285069870 0 0) 0x7f758413a740 con 0x7f75840b0860 2024-03-27T03:07:33.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75817fa640 10 monclient(hunting): _negotiate 2024-03-27T03:07:33.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75817fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:33.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:33.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75817fa640 10 monclient(hunting): my global_id is 4316 2024-03-27T03:07:33.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.255+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7570004eb0 con 0x7f75840b0860 2024-03-27T03:07:33.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2474773320 0 0) 0x7f7570004eb0 con 0x7f75840b0860 2024-03-27T03:07:33.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f758413a740 con 0x7f75840b0860 2024-03-27T03:07:33.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f75780017d0 con 0x7f75840b0860 2024-03-27T03:07:33.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:33.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:07:33.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 10 monclient(hunting): dump: 2024-03-27T03:07:33.250 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:33.250 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:33.250 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:33.250 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3871138856 0 0) 0x7f758413a740 con 0x7f75840b0860 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 >> v1:172.21.15.196:6789/0 conn(0x7f758413ef30 legacy=0x7f75841d59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 >> v1:172.21.15.153:6789/0 conn(0x7f75840af8e0 legacy=0x7f7584132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 10 monclient: _finish_hunting 0 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 1 monclient: found mon.c 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:33.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758413a8d0 con 0x7f75840b0860 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 10 monclient: _finish_auth 0 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 10 monclient: _check_auth_tickets 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7578003610 con 0x7f75840b0860 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f7578003920 con 0x7f75840b0860 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f758ab5d640 5 monclient: authenticate success, global_id 4316 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.256+0000 7f758ab5d640 10 monclient: _renew_subs 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.257+0000 7f758ab5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.257+0000 7f758ab5d640 1 -- 172.21.15.153:0/1441090443 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7584132e20 con 0x7f75840b0860 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.257+0000 7f758ab5d640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.257+0000 7f75817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.257+0000 7f75817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.257+0000 7f75817fa640 10 monclient: dump: 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:33.252 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:33.253 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:33.253 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:33.253 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:33.253 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:33.253 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:33.253 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:33.253 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:33.253 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:33.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.257+0000 7f758ab5d640 10 monclient: _renew_subs 2024-03-27T03:07:33.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.257+0000 7f758ab5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:33.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.257+0000 7f758ab5d640 1 -- 172.21.15.153:0/1441090443 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75840b13b0 con 0x7f75840b0860 2024-03-27T03:07:33.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.257+0000 7f758ab5d640 1 librados: init done 2024-03-27T03:07:33.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.258+0000 7f7582ffd640 4 set_mon_vals no callback set 2024-03-27T03:07:33.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.259+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f75780198f0 con 0x7f75840b0860 2024-03-27T03:07:33.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.260+0000 7f758ab5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:33.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.260+0000 7f758ab5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:33.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.260+0000 7f758ab5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:33.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.260+0000 7f758ab5d640 1 -- 172.21.15.153:0/1441090443 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7564005080 con 0x7f75840b0860 2024-03-27T03:07:33.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.261+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f7578083860 con 0x7f75840b0860 2024-03-27T03:07:33.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.261+0000 7f75817fa640 10 client.4316.objecter ms_dispatch 0x7f7584000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:07:33.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.261+0000 7f75817fa640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:07:33.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.261+0000 7f75817fa640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:07:33.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.261+0000 7f75817fa640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-27T03:07:33.255 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.263+0000 7f75888f9640 10 monclient: get_auth_request con 0x7f757006dcc0 auth_method 0 2024-03-27T03:07:33.257 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.265+0000 7f75817fa640 10 client.4316.objecter ms_handle_connect 0x7f757006dcc0 2024-03-27T03:07:33.262 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.269+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f757804f140 con 0x7f75840b0860 2024-03-27T03:07:33.262 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.269+0000 7f75817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:33.262 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.269+0000 7f75817fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:07:33.592 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.599+0000 7f758ab5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:33.592 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.599+0000 7f758ab5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:33.592 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.599+0000 7f758ab5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:33.592 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.599+0000 7f758ab5d640 1 -- 172.21.15.153:0/1441090443 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7564005330 con 0x7f75840b0860 2024-03-27T03:07:33.598 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:07:33.598 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:07:33.598 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.601+0000 7f75817fa640 1 -- 172.21.15.153:0/1441090443 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f7578051070 con 0x7f75840b0860 2024-03-27T03:07:33.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.601+0000 7f75817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:33.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.601+0000 7f75817fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:07:33.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.605+0000 7f758ab5d640 10 librados: watch_flush enter 2024-03-27T03:07:33.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.605+0000 7f758ab5d640 10 librados: watch_flush exit 2024-03-27T03:07:33.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.605+0000 7f758ab5d640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-27T03:07:33.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.605+0000 7f758ab5d640 10 client.4316.objecter successfully canceled tick 2024-03-27T03:07:33.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.607+0000 7f758ab5d640 1 -- 172.21.15.153:0/1441090443 >> v1:172.21.15.153:6816/2637937781 conn(0x7f757006dcc0 legacy=0x7f7570070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:33.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.607+0000 7f758ab5d640 10 monclient: shutdown 2024-03-27T03:07:33.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.607+0000 7f758ab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:33.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.608+0000 7f758ab5d640 1 -- 172.21.15.153:0/1441090443 >> v1:172.21.15.153:6790/0 conn(0x7f75840b0860 legacy=0x7f75841d52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:33.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.608+0000 7f758ab5d640 1 -- 172.21.15.153:0/1441090443 shutdown_connections 2024-03-27T03:07:33.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.608+0000 7f758ab5d640 1 -- 172.21.15.153:0/1441090443 >> 172.21.15.153:0/1441090443 conn(0x7f758406b660 msgr2=0x7f758413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:33.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.611+0000 7f758ab5d640 1 -- 172.21.15.153:0/1441090443 shutdown_connections 2024-03-27T03:07:33.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.611+0000 7f758ab5d640 1 -- 172.21.15.153:0/1441090443 wait complete. 2024-03-27T03:07:33.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:33.611+0000 7f758ab5d640 1 librados: shutdown 2024-03-27T03:07:33.623 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:07:40.430 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.599+0000 7f068f255640 10 monclient: get_monmap_and_config 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.599+0000 7f068f255640 10 monclient: build_initial_monmap 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.599+0000 7f068f255640 10 monclient: monmap: 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:07:40.600781+0000 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:07:40.600781+0000 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:07:40.593 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:40.595 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.602+0000 7f068f255640 1 Processor -- start 2024-03-27T03:07:40.595 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.602+0000 7f068f255640 1 -- start start 2024-03-27T03:07:40.595 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.602+0000 7f068f255640 10 monclient: init 2024-03-27T03:07:40.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.603+0000 7f068f255640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:40.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.603+0000 7f068f255640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:07:40.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.604+0000 7f068f255640 10 monclient(hunting): picked mon.noname-c con 0x7f06881457f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:40.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.604+0000 7f068f255640 10 monclient(hunting): picked mon.noname-a con 0x7f068813f650 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:40.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.604+0000 7f068f255640 10 monclient(hunting): picked mon.noname-b con 0x7f06880a9d70 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:40.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.604+0000 7f068f255640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06880aaef0 con 0x7f068813f650 2024-03-27T03:07:40.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.604+0000 7f068f255640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0688145bd0 con 0x7f06881457f0 2024-03-27T03:07:40.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.604+0000 7f068f255640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0688145eb0 con 0x7f06880a9d70 2024-03-27T03:07:40.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.604+0000 7f068f255640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:40.599 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.604+0000 7f068f255640 10 monclient(hunting): authenticate will time out at 1420.910327s 2024-03-27T03:07:40.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.607+0000 7f068cff1640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f06881457f0 0x7f068813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:43341/0 (socket says 172.21.15.153:43341) 2024-03-27T03:07:40.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.607+0000 7f068cff1640 1 -- 172.21.15.153:0/1953305738 learned_addr learned my addr 172.21.15.153:0/1953305738 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:40.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.608+0000 7f06877fe640 1 -- 172.21.15.153:0/1953305738 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971976914 0 0) 0x7f0688145bd0 con 0x7f06881457f0 2024-03-27T03:07:40.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.608+0000 7f06877fe640 10 monclient(hunting): _negotiate 2024-03-27T03:07:40.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.608+0000 7f06877fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:40.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.608+0000 7f06877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:40.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.608+0000 7f06877fe640 10 monclient(hunting): my global_id is 4309 2024-03-27T03:07:40.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.608+0000 7f06877fe640 1 -- 172.21.15.153:0/1953305738 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0670003520 con 0x7f06881457f0 2024-03-27T03:07:40.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.609+0000 7f06877fe640 1 -- 172.21.15.153:0/1953305738 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2687963875 0 0) 0x7f0670003520 con 0x7f06881457f0 2024-03-27T03:07:40.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.609+0000 7f06877fe640 1 -- 172.21.15.153:0/1953305738 >> v1:172.21.15.153:6790/0 conn(0x7f06880a9d70 legacy=0x7f068814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:07:40.603 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.610+0000 7f06877fe640 1 -- 172.21.15.153:0/1953305738 >> v1:172.21.15.153:6789/0 conn(0x7f068813f650 legacy=0x7f06880b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:07:40.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f06877fe640 10 monclient: _finish_hunting 0 2024-03-27T03:07:40.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f06877fe640 1 monclient: found mon.noname-c 2024-03-27T03:07:40.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f06877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:40.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f06877fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:07:40.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f06877fe640 1 -- 172.21.15.153:0/1953305738 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0688146340 con 0x7f06881457f0 2024-03-27T03:07:40.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f06877fe640 10 monclient: _finish_auth 0 2024-03-27T03:07:40.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f06877fe640 10 monclient: _check_auth_tickets 2024-03-27T03:07:40.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f06877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:40.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f06877fe640 1 -- 172.21.15.153:0/1953305738 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f067c001690 con 0x7f06881457f0 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f06877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f06877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f06877fe640 10 monclient: dump: 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f068f255640 5 monclient: authenticate success, global_id 4309 2024-03-27T03:07:40.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.612+0000 7f068f255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.613+0000 7f06877fe640 1 -- 172.21.15.153:0/1953305738 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f067c0018f0 con 0x7f06881457f0 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.613+0000 7f06877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.613+0000 7f06877fe640 1 -- 172.21.15.153:0/1953305738 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f067c001d40 con 0x7f06881457f0 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.613+0000 7f06877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.613+0000 7f06877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.613+0000 7f06877fe640 10 monclient: dump: 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:40.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.613+0000 7f068f255640 10 monclient: get_monmap_and_config success 2024-03-27T03:07:40.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.613+0000 7f068f255640 4 set_mon_vals no callback set 2024-03-27T03:07:40.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.614+0000 7f068f255640 10 monclient: shutdown 2024-03-27T03:07:40.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.614+0000 7f068f255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:40.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.614+0000 7f068f255640 1 -- 172.21.15.153:0/1953305738 >> v1:172.21.15.196:6789/0 conn(0x7f06881457f0 legacy=0x7f068813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:40.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.616+0000 7f068f255640 1 -- 172.21.15.153:0/1953305738 shutdown_connections 2024-03-27T03:07:40.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.616+0000 7f068f255640 1 -- 172.21.15.153:0/1953305738 >> 172.21.15.153:0/1953305738 conn(0x7f0688066a00 msgr2=0x7f0688066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:40.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.616+0000 7f068f255640 1 -- 172.21.15.153:0/1953305738 shutdown_connections 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.616+0000 7f068f255640 1 -- 172.21.15.153:0/1953305738 wait complete. 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.617+0000 7f068f255640 10 monclient: build_initial_monmap 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.617+0000 7f068f255640 10 monclient: monmap: 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.618+0000 7f068f255640 1 librados: starting msgr at 2024-03-27T03:07:40.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.618+0000 7f068f255640 1 librados: starting objecter 2024-03-27T03:07:40.613 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.618+0000 7f068f255640 1 Processor -- start 2024-03-27T03:07:40.613 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.618+0000 7f068f255640 1 -- start start 2024-03-27T03:07:40.613 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.618+0000 7f068f255640 1 librados: setting wanted keys 2024-03-27T03:07:40.613 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.618+0000 7f068f255640 1 librados: calling monclient init 2024-03-27T03:07:40.613 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.618+0000 7f068f255640 10 monclient: init 2024-03-27T03:07:40.613 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.619+0000 7f068f255640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:40.613 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.619+0000 7f068f255640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:07:40.617 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.623+0000 7f068f255640 10 monclient(hunting): picked mon.noname-c con 0x7f06880a9d70 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:40.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.625+0000 7f068f255640 10 monclient(hunting): picked mon.noname-b con 0x7f068813f650 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:40.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.625+0000 7f068f255640 10 monclient(hunting): picked mon.noname-a con 0x7f06881457f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:40.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.625+0000 7f068f255640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0688071930 con 0x7f06881457f0 2024-03-27T03:07:40.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.625+0000 7f068f255640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0688071c10 con 0x7f068813f650 2024-03-27T03:07:40.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.625+0000 7f068f255640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0688071ef0 con 0x7f06880a9d70 2024-03-27T03:07:40.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.625+0000 7f068f255640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:40.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.625+0000 7f068f255640 10 monclient(hunting): authenticate will time out at 1420.930883s 2024-03-27T03:07:40.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.626+0000 7f068cff1640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f06880a9d70 0x7f0688132250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:59131/0 (socket says 172.21.15.153:59131) 2024-03-27T03:07:40.620 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.626+0000 7f068cff1640 1 -- 172.21.15.153:0/2090071160 learned_addr learned my addr 172.21.15.153:0/2090071160 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:40.627 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.628+0000 7f06857fa640 10 client.?.objecter ms_handle_connect 0x7f06880a9d70 2024-03-27T03:07:40.627 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.628+0000 7f06857fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:40.627 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873640540 0 0) 0x7f0688071ef0 con 0x7f06880a9d70 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 10 monclient(hunting): _negotiate 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 10 monclient(hunting): my global_id is 4319 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0674004eb0 con 0x7f06880a9d70 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 10 client.?.objecter ms_handle_connect 0x7f068813f650 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1967277114 0 0) 0x7f0674004eb0 con 0x7f06880a9d70 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0688071ef0 con 0x7f06880a9d70 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f067c001650 con 0x7f06880a9d70 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 10 monclient(hunting): dump: 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:40.628 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.630+0000 7f06857fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3475769310 0 0) 0x7f0688071ef0 con 0x7f06880a9d70 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 >> v1:172.21.15.196:6789/0 conn(0x7f068813f650 legacy=0x7f0688132960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 >> v1:172.21.15.153:6789/0 conn(0x7f06881457f0 legacy=0x7f06881e5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 10 monclient: _finish_hunting 0 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 1 monclient: found mon.c 2024-03-27T03:07:40.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0688142320 con 0x7f06880a9d70 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 10 monclient: _finish_auth 0 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 10 monclient: _check_auth_tickets 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f068f255640 5 monclient: authenticate success, global_id 4319 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f068f255640 10 monclient: _renew_subs 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f068f255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f068f255640 1 -- 172.21.15.153:0/2090071160 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06881425f0 con 0x7f06880a9d70 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f068f255640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f068f255640 10 monclient: _renew_subs 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f068f255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f068f255640 1 -- 172.21.15.153:0/2090071160 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06880708b0 con 0x7f06880a9d70 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f068f255640 1 librados: init done 2024-03-27T03:07:40.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f067c004b10 con 0x7f06880a9d70 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f067c004ea0 con 0x7f06880a9d70 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.631+0000 7f06857fa640 10 monclient: dump: 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:40.631 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.632+0000 7f06867fc640 4 set_mon_vals no callback set 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.633+0000 7f068f255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.633+0000 7f068f255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.633+0000 7f068f255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.633+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f067c005000 con 0x7f06880a9d70 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.633+0000 7f068f255640 1 -- 172.21.15.153:0/2090071160 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0668005080 con 0x7f06880a9d70 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.634+0000 7f0687fff640 10 monclient: get_auth_request con 0x7f067406dc20 auth_method 0 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.634+0000 7f06857fa640 10 client.4319.objecter ms_handle_connect 0x7f067406dc20 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.634+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f067c084cc0 con 0x7f06880a9d70 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.634+0000 7f06857fa640 10 client.4319.objecter ms_dispatch 0x7f0688000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.634+0000 7f06857fa640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.635+0000 7f06857fa640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:07:40.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.635+0000 7f06857fa640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-27T03:07:40.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.642+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f067c0505a0 con 0x7f06880a9d70 2024-03-27T03:07:40.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.642+0000 7f06857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:40.636 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.642+0000 7f06857fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:07:40.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.961+0000 7f068f255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:40.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.961+0000 7f068f255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:40.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.961+0000 7f068f255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:07:40.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.961+0000 7f068f255640 1 -- 172.21.15.153:0/2090071160 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0668005330 con 0x7f06880a9d70 2024-03-27T03:07:40.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.962+0000 7f06857fa640 1 -- 172.21.15.153:0/2090071160 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f067c0524d0 con 0x7f06880a9d70 2024-03-27T03:07:40.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.962+0000 7f06857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:40.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.962+0000 7f06857fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:07:40.959 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:07:40.960 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:07:40.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.967+0000 7f068f255640 10 librados: watch_flush enter 2024-03-27T03:07:40.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.968+0000 7f068f255640 10 librados: watch_flush exit 2024-03-27T03:07:40.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.968+0000 7f068f255640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-27T03:07:40.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.968+0000 7f068f255640 10 client.4319.objecter successfully canceled tick 2024-03-27T03:07:40.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.968+0000 7f068f255640 1 -- 172.21.15.153:0/2090071160 >> v1:172.21.15.153:6816/2637937781 conn(0x7f067406dc20 legacy=0x7f06740700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:40.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.968+0000 7f068f255640 10 monclient: shutdown 2024-03-27T03:07:40.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.968+0000 7f068f255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:40.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.968+0000 7f068f255640 1 -- 172.21.15.153:0/2090071160 >> v1:172.21.15.153:6790/0 conn(0x7f06880a9d70 legacy=0x7f0688132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:40.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.968+0000 7f068f255640 1 -- 172.21.15.153:0/2090071160 shutdown_connections 2024-03-27T03:07:40.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.968+0000 7f068f255640 1 -- 172.21.15.153:0/2090071160 >> 172.21.15.153:0/2090071160 conn(0x7f0688066a00 msgr2=0x7f06880b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:40.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.969+0000 7f068f255640 1 -- 172.21.15.153:0/2090071160 shutdown_connections 2024-03-27T03:07:40.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.969+0000 7f068f255640 1 -- 172.21.15.153:0/2090071160 wait complete. 2024-03-27T03:07:40.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:40.969+0000 7f068f255640 1 librados: shutdown 2024-03-27T03:07:40.978 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:07:47.784 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:07:47.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.941+0000 7f1e82001640 10 monclient: get_monmap_and_config 2024-03-27T03:07:47.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.941+0000 7f1e82001640 10 monclient: build_initial_monmap 2024-03-27T03:07:47.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.941+0000 7f1e82001640 10 monclient: monmap: 2024-03-27T03:07:47.935 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:47.935 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:47.935 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:07:47.942503+0000 2024-03-27T03:07:47.935 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:07:47.942503+0000 2024-03-27T03:07:47.935 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:47.935 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:47.935 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:47.935 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:07:47.935 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:07:47.935 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:47.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.942+0000 7f1e82001640 1 Processor -- start 2024-03-27T03:07:47.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.943+0000 7f1e82001640 1 -- start start 2024-03-27T03:07:47.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.943+0000 7f1e82001640 10 monclient: init 2024-03-27T03:07:47.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.943+0000 7f1e82001640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:47.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.943+0000 7f1e82001640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:07:47.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.944+0000 7f1e82001640 10 monclient(hunting): picked mon.noname-b con 0x7f1e7c0b4330 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:47.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.944+0000 7f1e82001640 10 monclient(hunting): picked mon.noname-c con 0x7f1e7c1470b0 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:47.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.944+0000 7f1e82001640 10 monclient(hunting): picked mon.noname-a con 0x7f1e7c1439b0 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:47.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.944+0000 7f1e82001640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c06f2a0 con 0x7f1e7c1439b0 2024-03-27T03:07:47.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.944+0000 7f1e82001640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c146b90 con 0x7f1e7c1470b0 2024-03-27T03:07:47.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.944+0000 7f1e82001640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c0b2ca0 con 0x7f1e7c0b4330 2024-03-27T03:07:47.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.944+0000 7f1e82001640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:47.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.944+0000 7f1e82001640 10 monclient(hunting): authenticate will time out at 1428.249660s 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.944+0000 7f1e7bfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1e7c1439b0 0x7f1e7c151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:52648/0 (socket says 172.21.15.153:52648) 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.944+0000 7f1e7bfff640 1 -- 172.21.15.153:0/3122076014 learned_addr learned my addr 172.21.15.153:0/3122076014 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735732488 0 0) 0x7f1e7c146b90 con 0x7f1e7c1470b0 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 10 monclient(hunting): my global_id is 4315 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e64003520 con 0x7f1e7c1470b0 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015010984 0 0) 0x7f1e7c06f2a0 con 0x7f1e7c1439b0 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 10 monclient(hunting): my global_id is 4299 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e7c146b90 con 0x7f1e7c1439b0 2024-03-27T03:07:47.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960917260 0 0) 0x7f1e7c0b2ca0 con 0x7f1e7c0b4330 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 10 monclient(hunting): my global_id is 4322 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.945+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e7c06f2a0 con 0x7f1e7c0b4330 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 467112472 0 0) 0x7f1e64003520 con 0x7f1e7c1470b0 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 >> v1:172.21.15.153:6790/0 conn(0x7f1e7c0b4330 legacy=0x7f1e7c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 >> v1:172.21.15.153:6789/0 conn(0x7f1e7c1439b0 legacy=0x7f1e7c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 1 monclient: found mon.noname-c 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c06e0c0 con 0x7f1e7c1470b0 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 10 monclient: _finish_auth 0 2024-03-27T03:07:47.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f1e6c001690 con 0x7f1e7c1470b0 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e82001640 5 monclient: authenticate success, global_id 4315 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e82001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 10 monclient: dump: 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:47.945 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e82001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e6c0017f0 con 0x7f1e7c1470b0 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 1 -- 172.21.15.153:0/3122076014 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f1e6c001c00 con 0x7f1e7c1470b0 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e7a7fc640 10 monclient: dump: 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:47.946 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e82001640 10 monclient: get_monmap_and_config success 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.946+0000 7f1e82001640 4 set_mon_vals no callback set 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.947+0000 7f1e82001640 10 monclient: shutdown 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.947+0000 7f1e82001640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.947+0000 7f1e82001640 1 -- 172.21.15.153:0/3122076014 >> v1:172.21.15.196:6789/0 conn(0x7f1e7c1470b0 legacy=0x7f1e7c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.947+0000 7f1e82001640 1 -- 172.21.15.153:0/3122076014 shutdown_connections 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.947+0000 7f1e82001640 1 -- 172.21.15.153:0/3122076014 >> 172.21.15.153:0/3122076014 conn(0x7f1e7c066a00 msgr2=0x7f1e7c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.948+0000 7f1e82001640 1 -- 172.21.15.153:0/3122076014 shutdown_connections 2024-03-27T03:07:47.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.948+0000 7f1e82001640 1 -- 172.21.15.153:0/3122076014 wait complete. 2024-03-27T03:07:47.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.948+0000 7f1e82001640 10 monclient: build_initial_monmap 2024-03-27T03:07:47.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.948+0000 7f1e82001640 10 monclient: monmap: 2024-03-27T03:07:47.954 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.950+0000 7f1e82001640 1 librados: starting msgr at 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.950+0000 7f1e82001640 1 librados: starting objecter 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.951+0000 7f1e82001640 1 Processor -- start 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.951+0000 7f1e82001640 1 -- start start 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.951+0000 7f1e82001640 1 librados: setting wanted keys 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.951+0000 7f1e82001640 1 librados: calling monclient init 2024-03-27T03:07:47.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.951+0000 7f1e82001640 10 monclient: init 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e82001640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e82001640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e82001640 10 monclient(hunting): picked mon.noname-c con 0x7f1e7c0b4330 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e82001640 10 monclient(hunting): picked mon.noname-a con 0x7f1e7c1439b0 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e82001640 10 monclient(hunting): picked mon.noname-b con 0x7f1e7c1470b0 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e82001640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c1e8280 con 0x7f1e7c1439b0 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e82001640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c0b13a0 con 0x7f1e7c1470b0 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e82001640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c142930 con 0x7f1e7c0b4330 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e82001640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e82001640 10 monclient(hunting): authenticate will time out at 1428.257995s 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e7affd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1e7c1439b0 0x7f1e7c141c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:52936/0 (socket says 172.21.15.153:52936) 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e7affd640 1 -- 172.21.15.153:0/2548242355 learned_addr learned my addr 172.21.15.153:0/2548242355 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e63fff640 10 client.?.objecter ms_handle_connect 0x7f1e7c1439b0 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.952+0000 7f1e63fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 client.?.objecter ms_handle_connect 0x7f1e7c1470b0 2024-03-27T03:07:47.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 client.?.objecter ms_handle_connect 0x7f1e7c0b4330 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998043476 0 0) 0x7f1e7c1e8280 con 0x7f1e7c1439b0 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 monclient(hunting): _negotiate 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 monclient(hunting): my global_id is 4302 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e68004eb0 con 0x7f1e7c1439b0 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806131209 0 0) 0x7f1e7c0b13a0 con 0x7f1e7c1470b0 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 monclient(hunting): _negotiate 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 monclient(hunting): my global_id is 4318 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e7c1e8280 con 0x7f1e7c1470b0 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262228638 0 0) 0x7f1e7c142930 con 0x7f1e7c0b4330 2024-03-27T03:07:47.957 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 monclient(hunting): _negotiate 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 10 monclient(hunting): my global_id is 4325 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.953+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e7c0b13a0 con 0x7f1e7c0b4330 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3916504575 0 0) 0x7f1e7c1e8280 con 0x7f1e7c1470b0 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e68005620 con 0x7f1e7c1470b0 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 133805616 0 0) 0x7f1e68004eb0 con 0x7f1e7c1439b0 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e7c1e8280 con 0x7f1e7c1439b0 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f1e74001cb0 con 0x7f1e7c1470b0 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 10 monclient(hunting): dump: 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:47.958 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f1e6c001b40 con 0x7f1e7c1439b0 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 10 monclient(hunting): dump: 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:47.959 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1423065497 0 0) 0x7f1e68005620 con 0x7f1e7c1470b0 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 >> v1:172.21.15.153:6790/0 conn(0x7f1e7c0b4330 legacy=0x7f1e7c1e7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 >> v1:172.21.15.153:6789/0 conn(0x7f1e7c1439b0 legacy=0x7f1e7c141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 10 monclient: _finish_hunting 0 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 1 monclient: found mon.b 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c1e7da0 con 0x7f1e7c1470b0 2024-03-27T03:07:47.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 10 monclient: _finish_auth 0 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 10 monclient: _check_auth_tickets 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e82001640 5 monclient: authenticate success, global_id 4318 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e82001640 10 monclient: _renew_subs 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e82001640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.954+0000 7f1e82001640 1 -- 172.21.15.153:0/2548242355 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e7c1e7ff0 con 0x7f1e7c1470b0 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.955+0000 7f1e82001640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.955+0000 7f1e82001640 10 monclient: _renew_subs 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.955+0000 7f1e82001640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.955+0000 7f1e82001640 1 -- 172.21.15.153:0/2548242355 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7c142540 con 0x7f1e7c1470b0 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.955+0000 7f1e82001640 1 librados: init done 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.955+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e74004b60 con 0x7f1e7c1470b0 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.955+0000 7f1e63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.955+0000 7f1e79ffb640 4 set_mon_vals no callback set 2024-03-27T03:07:47.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.955+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f1e74004e70 con 0x7f1e7c1470b0 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.955+0000 7f1e63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.955+0000 7f1e63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.955+0000 7f1e63fff640 10 monclient: dump: 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.957+0000 7f1e82001640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:47.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.957+0000 7f1e82001640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:47.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.957+0000 7f1e82001640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:47.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.957+0000 7f1e82001640 1 -- 172.21.15.153:0/2548242355 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5c005080 con 0x7f1e7c1470b0 2024-03-27T03:07:47.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.958+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f1e7401ae40 con 0x7f1e7c1470b0 2024-03-27T03:07:47.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.959+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f1e74084cf0 con 0x7f1e7c1470b0 2024-03-27T03:07:47.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.959+0000 7f1e63fff640 10 client.4318.objecter ms_dispatch 0x7f1e7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:07:47.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.959+0000 7f1e63fff640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:07:47.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.959+0000 7f1e63fff640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:07:47.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.959+0000 7f1e63fff640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-27T03:07:47.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.960+0000 7f1e7b7fe640 10 monclient: get_auth_request con 0x7f1e6806e1d0 auth_method 0 2024-03-27T03:07:47.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.960+0000 7f1e63fff640 10 client.4318.objecter ms_handle_connect 0x7f1e6806e1d0 2024-03-27T03:07:47.964 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.965+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e740505d0 con 0x7f1e7c1470b0 2024-03-27T03:07:47.964 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.965+0000 7f1e63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:47.964 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:47.965+0000 7f1e63fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:07:48.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.330+0000 7f1e82001640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:48.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.330+0000 7f1e82001640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:48.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.330+0000 7f1e82001640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:48.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.330+0000 7f1e82001640 1 -- 172.21.15.153:0/2548242355 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e5c005330 con 0x7f1e7c1470b0 2024-03-27T03:07:48.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.331+0000 7f1e63fff640 1 -- 172.21.15.153:0/2548242355 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f1e74052500 con 0x7f1e7c1470b0 2024-03-27T03:07:48.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.331+0000 7f1e63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:48.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.331+0000 7f1e63fff640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:07:48.324 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:07:48.324 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:07:48.327 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.334+0000 7f1e82001640 10 librados: watch_flush enter 2024-03-27T03:07:48.328 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.335+0000 7f1e82001640 10 librados: watch_flush exit 2024-03-27T03:07:48.328 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.335+0000 7f1e82001640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-27T03:07:48.328 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.335+0000 7f1e82001640 10 client.4318.objecter successfully canceled tick 2024-03-27T03:07:48.328 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.335+0000 7f1e82001640 1 -- 172.21.15.153:0/2548242355 >> v1:172.21.15.153:6816/2637937781 conn(0x7f1e6806e1d0 legacy=0x7f1e68070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:48.328 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.335+0000 7f1e82001640 10 monclient: shutdown 2024-03-27T03:07:48.328 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.335+0000 7f1e82001640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:48.328 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.335+0000 7f1e82001640 1 -- 172.21.15.153:0/2548242355 >> v1:172.21.15.196:6789/0 conn(0x7f1e7c1470b0 legacy=0x7f1e7c0ac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:48.329 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.335+0000 7f1e82001640 1 -- 172.21.15.153:0/2548242355 shutdown_connections 2024-03-27T03:07:48.329 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.335+0000 7f1e82001640 1 -- 172.21.15.153:0/2548242355 >> 172.21.15.153:0/2548242355 conn(0x7f1e7c066a00 msgr2=0x7f1e7c145ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:48.329 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.335+0000 7f1e82001640 1 -- 172.21.15.153:0/2548242355 shutdown_connections 2024-03-27T03:07:48.329 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.335+0000 7f1e82001640 1 -- 172.21.15.153:0/2548242355 wait complete. 2024-03-27T03:07:48.329 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:48.336+0000 7f1e82001640 1 librados: shutdown 2024-03-27T03:07:48.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:07:55.153 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:07:55.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.303+0000 7fe1213b6640 10 monclient: get_monmap_and_config 2024-03-27T03:07:55.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.303+0000 7fe1213b6640 10 monclient: build_initial_monmap 2024-03-27T03:07:55.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.303+0000 7fe1213b6640 10 monclient: monmap: 2024-03-27T03:07:55.296 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:55.296 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:55.296 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:07:55.304141+0000 2024-03-27T03:07:55.296 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:07:55.304141+0000 2024-03-27T03:07:55.296 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:55.297 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:55.297 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:55.297 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:07:55.297 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:07:55.297 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:55.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.304+0000 7fe1213b6640 1 Processor -- start 2024-03-27T03:07:55.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.304+0000 7fe1213b6640 1 -- start start 2024-03-27T03:07:55.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.304+0000 7fe1213b6640 10 monclient: init 2024-03-27T03:07:55.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.305+0000 7fe1213b6640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:55.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.305+0000 7fe1213b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:07:55.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.305+0000 7fe1213b6640 10 monclient(hunting): picked mon.noname-c con 0x7fe11c1457f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:55.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.305+0000 7fe1213b6640 10 monclient(hunting): picked mon.noname-a con 0x7fe11c13f650 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:55.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.305+0000 7fe1213b6640 10 monclient(hunting): picked mon.noname-b con 0x7fe11c0a9d70 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:55.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.305+0000 7fe1213b6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c0aaef0 con 0x7fe11c13f650 2024-03-27T03:07:55.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.305+0000 7fe1213b6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c145bd0 con 0x7fe11c1457f0 2024-03-27T03:07:55.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.305+0000 7fe1213b6640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c145eb0 con 0x7fe11c0a9d70 2024-03-27T03:07:55.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.305+0000 7fe1213b6640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:55.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.305+0000 7fe1213b6640 10 monclient(hunting): authenticate will time out at 1435.611280s 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.306+0000 7fe11a7fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe11c13f650 0x7fe11c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:34108/0 (socket says 172.21.15.153:34108) 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.306+0000 7fe11a7fc640 1 -- 172.21.15.153:0/2334508904 learned_addr learned my addr 172.21.15.153:0/2334508904 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.306+0000 7fe119ffb640 1 -- 172.21.15.153:0/2334508904 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557828517 0 0) 0x7fe11c145eb0 con 0x7fe11c0a9d70 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.306+0000 7fe119ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.306+0000 7fe119ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.306+0000 7fe119ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.306+0000 7fe119ffb640 10 monclient(hunting): my global_id is 4328 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.306+0000 7fe119ffb640 1 -- 172.21.15.153:0/2334508904 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe108003520 con 0x7fe11c0a9d70 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 1 -- 172.21.15.153:0/2334508904 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537875615 0 0) 0x7fe11c0aaef0 con 0x7fe11c13f650 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient(hunting): my global_id is 4305 2024-03-27T03:07:55.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 1 -- 172.21.15.153:0/2334508904 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe11c145eb0 con 0x7fe11c13f650 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 1 -- 172.21.15.153:0/2334508904 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2768094377 0 0) 0x7fe108003520 con 0x7fe11c0a9d70 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 1 -- 172.21.15.153:0/2334508904 >> v1:172.21.15.196:6789/0 conn(0x7fe11c1457f0 legacy=0x7fe11c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 1 -- 172.21.15.153:0/2334508904 >> v1:172.21.15.153:6789/0 conn(0x7fe11c13f650 legacy=0x7fe11c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 1 monclient: found mon.noname-b 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 1 -- 172.21.15.153:0/2334508904 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c146340 con 0x7fe11c0a9d70 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient: _finish_auth 0 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 1 -- 172.21.15.153:0/2334508904 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe114001a10 con 0x7fe11c0a9d70 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:07:55.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient: dump: 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe1213b6640 5 monclient: authenticate success, global_id 4328 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe1213b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 1 -- 172.21.15.153:0/2334508904 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe114001b70 con 0x7fe11c0a9d70 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:55.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 1 -- 172.21.15.153:0/2334508904 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe114001f80 con 0x7fe11c0a9d70 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe119ffb640 10 monclient: dump: 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:55.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe1213b6640 10 monclient: get_monmap_and_config success 2024-03-27T03:07:55.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.307+0000 7fe1213b6640 4 set_mon_vals no callback set 2024-03-27T03:07:55.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.308+0000 7fe1213b6640 10 monclient: shutdown 2024-03-27T03:07:55.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.308+0000 7fe1213b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:55.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.308+0000 7fe1213b6640 1 -- 172.21.15.153:0/2334508904 >> v1:172.21.15.153:6790/0 conn(0x7fe11c0a9d70 legacy=0x7fe11c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:55.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.308+0000 7fe1213b6640 1 -- 172.21.15.153:0/2334508904 shutdown_connections 2024-03-27T03:07:55.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.308+0000 7fe1213b6640 1 -- 172.21.15.153:0/2334508904 >> 172.21.15.153:0/2334508904 conn(0x7fe11c066a00 msgr2=0x7fe11c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:55.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.308+0000 7fe1213b6640 1 -- 172.21.15.153:0/2334508904 shutdown_connections 2024-03-27T03:07:55.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.308+0000 7fe1213b6640 1 -- 172.21.15.153:0/2334508904 wait complete. 2024-03-27T03:07:55.314 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.309+0000 7fe1213b6640 10 monclient: build_initial_monmap 2024-03-27T03:07:55.314 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.309+0000 7fe1213b6640 10 monclient: monmap: 2024-03-27T03:07:55.314 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:07:55.314 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:07:55.314 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:07:55.314 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:07:55.314 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:07:55.314 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:55.314 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:07:55.314 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.310+0000 7fe1213b6640 1 librados: starting msgr at 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.310+0000 7fe1213b6640 1 librados: starting objecter 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.310+0000 7fe1213b6640 1 Processor -- start 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.310+0000 7fe1213b6640 1 -- start start 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.310+0000 7fe1213b6640 1 librados: setting wanted keys 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.310+0000 7fe1213b6640 1 librados: calling monclient init 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.310+0000 7fe1213b6640 10 monclient: init 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1213b6640 10 monclient: _reopen_session rank -1 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1213b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1213b6640 10 monclient(hunting): picked mon.noname-c con 0x7fe11c0a9d70 addr v1:172.21.15.153:6790/0 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1213b6640 10 monclient(hunting): picked mon.noname-b con 0x7fe11c13f650 addr v1:172.21.15.196:6789/0 2024-03-27T03:07:55.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1213b6640 10 monclient(hunting): picked mon.noname-a con 0x7fe11c1457f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1213b6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c06fb60 con 0x7fe11c1457f0 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1213b6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c0711b0 con 0x7fe11c13f650 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1213b6640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c071490 con 0x7fe11c0a9d70 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1213b6640 10 monclient(hunting): _renew_subs 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1213b6640 10 monclient(hunting): authenticate will time out at 1435.617112s 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe11a7fc640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe11c13f650 0x7fe11c139830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:50838/0 (socket says 172.21.15.153:50838) 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe11a7fc640 1 -- 172.21.15.153:0/563575632 learned_addr learned my addr 172.21.15.153:0/563575632 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1137fe640 10 client.?.objecter ms_handle_connect 0x7fe11c13f650 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1137fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1137fe640 10 client.?.objecter ms_handle_connect 0x7fe11c0a9d70 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1137fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.311+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428029042 0 0) 0x7fe11c0711b0 con 0x7fe11c13f650 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 monclient(hunting): _negotiate 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:55.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 monclient(hunting): my global_id is 4321 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0fc004eb0 con 0x7fe11c13f650 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547161941 0 0) 0x7fe11c071490 con 0x7fe11c0a9d70 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 monclient(hunting): _negotiate 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 monclient(hunting): my global_id is 4331 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe11c0711b0 con 0x7fe11c0a9d70 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1341342761 0 0) 0x7fe0fc004eb0 con 0x7fe11c13f650 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe11c071490 con 0x7fe11c13f650 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 client.?.objecter ms_handle_connect 0x7fe11c1457f0 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe1040019f0 con 0x7fe11c13f650 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:55.317 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 monclient(hunting): dump: 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.312+0000 7fe1137fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3058872451 0 0) 0x7fe11c071490 con 0x7fe11c13f650 2024-03-27T03:07:55.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 >> v1:172.21.15.153:6790/0 conn(0x7fe11c0a9d70 legacy=0x7fe11c1db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 >> v1:172.21.15.153:6789/0 conn(0x7fe11c1457f0 legacy=0x7fe11c1e6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 10 monclient: _finish_hunting 0 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 1 monclient: found mon.b 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c06f760 con 0x7fe11c13f650 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 10 monclient: _finish_auth 0 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 10 monclient: _check_auth_tickets 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1213b6640 5 monclient: authenticate success, global_id 4321 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1213b6640 10 monclient: _renew_subs 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1213b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1213b6640 1 -- 172.21.15.153:0/563575632 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c06fff0 con 0x7fe11c13f650 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1213b6640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:07:55.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1213b6640 10 monclient: _renew_subs 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1213b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1213b6640 1 -- 172.21.15.153:0/563575632 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c06f010 con 0x7fe11c13f650 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1213b6640 1 librados: init done 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1040047e0 con 0x7fe11c13f650 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe104004af0 con 0x7fe11c13f650 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1137fe640 10 monclient: dump: 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:07:55.320 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.313+0000 7fe1197fa640 4 set_mon_vals no callback set 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.315+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fe104005cc0 con 0x7fe11c13f650 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.315+0000 7fe1213b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.315+0000 7fe1213b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.315+0000 7fe1213b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.315+0000 7fe1213b6640 1 -- 172.21.15.153:0/563575632 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11c075f10 con 0x7fe11c13f650 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.316+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fe104084b70 con 0x7fe11c13f650 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.316+0000 7fe1137fe640 10 client.4321.objecter ms_dispatch 0x7fe11c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.316+0000 7fe1137fe640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.316+0000 7fe1137fe640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:07:55.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.316+0000 7fe1137fe640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-27T03:07:55.322 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.316+0000 7fe11affd640 10 monclient: get_auth_request con 0x7fe0fc06dd10 auth_method 0 2024-03-27T03:07:55.322 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.316+0000 7fe1137fe640 10 client.4321.objecter ms_handle_connect 0x7fe0fc06dd10 2024-03-27T03:07:55.322 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.326+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe104050450 con 0x7fe11c13f650 2024-03-27T03:07:55.322 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.326+0000 7fe1137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:07:55.322 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.326+0000 7fe1137fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:07:55.692 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.699+0000 7fe1213b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:55.692 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.699+0000 7fe1213b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:55.692 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.699+0000 7fe1213b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:07:55.692 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.699+0000 7fe1213b6640 1 -- 172.21.15.153:0/563575632 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe11c0b3680 con 0x7fe11c13f650 2024-03-27T03:07:55.692 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.700+0000 7fe1137fe640 1 -- 172.21.15.153:0/563575632 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fe104052380 con 0x7fe11c13f650 2024-03-27T03:07:55.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.700+0000 7fe1137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:07:55.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.700+0000 7fe1137fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:07:55.693 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:07:55.693 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:07:55.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.703+0000 7fe1213b6640 10 librados: watch_flush enter 2024-03-27T03:07:55.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.703+0000 7fe1213b6640 10 librados: watch_flush exit 2024-03-27T03:07:55.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.703+0000 7fe1213b6640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-27T03:07:55.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.703+0000 7fe1213b6640 10 client.4321.objecter successfully canceled tick 2024-03-27T03:07:55.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.704+0000 7fe1213b6640 1 -- 172.21.15.153:0/563575632 >> v1:172.21.15.153:6816/2637937781 conn(0x7fe0fc06dd10 legacy=0x7fe0fc0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:55.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.704+0000 7fe1213b6640 10 monclient: shutdown 2024-03-27T03:07:55.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.704+0000 7fe1213b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:07:55.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.704+0000 7fe1213b6640 1 -- 172.21.15.153:0/563575632 >> v1:172.21.15.196:6789/0 conn(0x7fe11c13f650 legacy=0x7fe11c139830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:07:55.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.704+0000 7fe1213b6640 1 -- 172.21.15.153:0/563575632 shutdown_connections 2024-03-27T03:07:55.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.704+0000 7fe1213b6640 1 -- 172.21.15.153:0/563575632 >> 172.21.15.153:0/563575632 conn(0x7fe11c066a00 msgr2=0x7fe11c0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:07:55.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.704+0000 7fe1213b6640 1 -- 172.21.15.153:0/563575632 shutdown_connections 2024-03-27T03:07:55.698 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.704+0000 7fe1213b6640 1 -- 172.21.15.153:0/563575632 wait complete. 2024-03-27T03:07:55.698 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:07:55.704+0000 7fe1213b6640 1 librados: shutdown 2024-03-27T03:07:55.712 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:08:02.519 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:08:02.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.674+0000 7fe996bb4640 10 monclient: get_monmap_and_config 2024-03-27T03:08:02.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.674+0000 7fe996bb4640 10 monclient: build_initial_monmap 2024-03-27T03:08:02.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.674+0000 7fe996bb4640 10 monclient: monmap: 2024-03-27T03:08:02.669 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:02.669 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:02.669 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:08:02.676378+0000 2024-03-27T03:08:02.669 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:08:02.676378+0000 2024-03-27T03:08:02.669 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:02.669 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:02.669 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:02.669 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:08:02.669 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:08:02.669 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:02.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.676+0000 7fe996bb4640 1 Processor -- start 2024-03-27T03:08:02.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.676+0000 7fe996bb4640 1 -- start start 2024-03-27T03:08:02.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.676+0000 7fe996bb4640 10 monclient: init 2024-03-27T03:08:02.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.677+0000 7fe996bb4640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:02.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.677+0000 7fe996bb4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:08:02.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.677+0000 7fe996bb4640 10 monclient(hunting): picked mon.noname-b con 0x7fe9880389e0 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:02.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.677+0000 7fe996bb4640 10 monclient(hunting): picked mon.noname-c con 0x7fe988038dc0 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:02.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.677+0000 7fe996bb4640 10 monclient(hunting): picked mon.noname-a con 0x7fe988030660 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:02.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.677+0000 7fe996bb4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9880c54c0 con 0x7fe988030660 2024-03-27T03:08:02.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.677+0000 7fe996bb4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe988036430 con 0x7fe988038dc0 2024-03-27T03:08:02.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.677+0000 7fe996bb4640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe988039840 con 0x7fe9880389e0 2024-03-27T03:08:02.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.677+0000 7fe996bb4640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:02.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.677+0000 7fe996bb4640 10 monclient(hunting): authenticate will time out at 1442.984101s 2024-03-27T03:08:02.676 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe995bb2640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fe9880389e0 0x7fe988033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:41099/0 (socket says 172.21.15.153:41099) 2024-03-27T03:08:02.676 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe995bb2640 1 -- 172.21.15.153:0/2555612849 learned_addr learned my addr 172.21.15.153:0/2555612849 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:02.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe994bb0640 1 -- 172.21.15.153:0/2555612849 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930482000 0 0) 0x7fe9880c54c0 con 0x7fe988030660 2024-03-27T03:08:02.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe994bb0640 10 monclient(hunting): _negotiate 2024-03-27T03:08:02.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe994bb0640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:02.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe994bb0640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:02.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe994bb0640 10 monclient(hunting): my global_id is 4311 2024-03-27T03:08:02.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe994bb0640 1 -- 172.21.15.153:0/2555612849 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe97c003520 con 0x7fe988030660 2024-03-27T03:08:02.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe994bb0640 1 -- 172.21.15.153:0/2555612849 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181498618 0 0) 0x7fe988039840 con 0x7fe9880389e0 2024-03-27T03:08:02.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe994bb0640 10 monclient(hunting): _negotiate 2024-03-27T03:08:02.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe994bb0640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe994bb0640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe994bb0640 10 monclient(hunting): my global_id is 4334 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.682+0000 7fe994bb0640 1 -- 172.21.15.153:0/2555612849 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9880c54c0 con 0x7fe9880389e0 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 1 -- 172.21.15.153:0/2555612849 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2615307063 0 0) 0x7fe9880c54c0 con 0x7fe9880389e0 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 1 -- 172.21.15.153:0/2555612849 >> v1:172.21.15.196:6789/0 conn(0x7fe988038dc0 legacy=0x7fe9880cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 1 -- 172.21.15.153:0/2555612849 >> v1:172.21.15.153:6789/0 conn(0x7fe988030660 legacy=0x7fe9880d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 10 monclient: _finish_hunting 0 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 1 monclient: found mon.noname-b 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 1 -- 172.21.15.153:0/2555612849 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe988035690 con 0x7fe9880389e0 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 10 monclient: _finish_auth 0 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 10 monclient: _check_auth_tickets 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 1 -- 172.21.15.153:0/2555612849 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe98c001610 con 0x7fe9880389e0 2024-03-27T03:08:02.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 10 monclient: dump: 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 1 -- 172.21.15.153:0/2555612849 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe98c0018a0 con 0x7fe9880389e0 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:02.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 1 -- 172.21.15.153:0/2555612849 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe98c001c30 con 0x7fe9880389e0 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe994bb0640 10 monclient: dump: 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe996bb4640 5 monclient: authenticate success, global_id 4334 2024-03-27T03:08:02.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe996bb4640 10 monclient: get_monmap_and_config success 2024-03-27T03:08:02.682 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.683+0000 7fe996bb4640 4 set_mon_vals no callback set 2024-03-27T03:08:02.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.684+0000 7fe996bb4640 10 monclient: shutdown 2024-03-27T03:08:02.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.684+0000 7fe996bb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:02.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.684+0000 7fe996bb4640 1 -- 172.21.15.153:0/2555612849 >> v1:172.21.15.153:6790/0 conn(0x7fe9880389e0 legacy=0x7fe988033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:02.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.684+0000 7fe996bb4640 1 -- 172.21.15.153:0/2555612849 shutdown_connections 2024-03-27T03:08:02.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.684+0000 7fe996bb4640 1 -- 172.21.15.153:0/2555612849 >> 172.21.15.153:0/2555612849 conn(0x7fe988005c00 msgr2=0x7fe988006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:02.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.685+0000 7fe996bb4640 1 -- 172.21.15.153:0/2555612849 shutdown_connections 2024-03-27T03:08:02.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.685+0000 7fe996bb4640 1 -- 172.21.15.153:0/2555612849 wait complete. 2024-03-27T03:08:02.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.686+0000 7fe996bb4640 10 monclient: build_initial_monmap 2024-03-27T03:08:02.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.686+0000 7fe996bb4640 10 monclient: monmap: 2024-03-27T03:08:02.684 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:02.684 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.686+0000 7fe996bb4640 1 librados: starting msgr at 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.686+0000 7fe996bb4640 1 librados: starting objecter 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.686+0000 7fe996bb4640 1 Processor -- start 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.686+0000 7fe996bb4640 1 -- start start 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.686+0000 7fe996bb4640 1 librados: setting wanted keys 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.686+0000 7fe996bb4640 1 librados: calling monclient init 2024-03-27T03:08:02.685 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.686+0000 7fe996bb4640 10 monclient: init 2024-03-27T03:08:02.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.687+0000 7fe996bb4640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:02.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.687+0000 7fe996bb4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:08:02.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.687+0000 7fe996bb4640 10 monclient(hunting): picked mon.noname-b con 0x7fe988030660 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:02.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.687+0000 7fe996bb4640 10 monclient(hunting): picked mon.noname-c con 0x7fe9880389e0 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:02.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.687+0000 7fe996bb4640 10 monclient(hunting): picked mon.noname-a con 0x7fe98816d3e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:02.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.687+0000 7fe996bb4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98802e4b0 con 0x7fe98816d3e0 2024-03-27T03:08:02.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.687+0000 7fe996bb4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98802e700 con 0x7fe988030660 2024-03-27T03:08:02.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.687+0000 7fe996bb4640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9880c7890 con 0x7fe9880389e0 2024-03-27T03:08:02.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.687+0000 7fe996bb4640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:02.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.687+0000 7fe996bb4640 10 monclient(hunting): authenticate will time out at 1442.994226s 2024-03-27T03:08:02.687 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.691+0000 7fe995bb2640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe988030660 0x7fe98816ccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:37193/0 (socket says 172.21.15.153:37193) 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.692+0000 7fe995bb2640 1 -- 172.21.15.153:0/326397142 learned_addr learned my addr 172.21.15.153:0/326397142 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.692+0000 7fe9867fc640 10 client.?.objecter ms_handle_connect 0x7fe988030660 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.692+0000 7fe9867fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.692+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423797580 0 0) 0x7fe98802e700 con 0x7fe988030660 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.692+0000 7fe9867fc640 10 monclient(hunting): _negotiate 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.692+0000 7fe9867fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.692+0000 7fe9867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.692+0000 7fe9867fc640 10 monclient(hunting): my global_id is 4327 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.692+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe974004eb0 con 0x7fe988030660 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.693+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1094183932 0 0) 0x7fe974004eb0 con 0x7fe988030660 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.693+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe98802e700 con 0x7fe988030660 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.693+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe98c0014d0 con 0x7fe988030660 2024-03-27T03:08:02.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.693+0000 7fe9867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.693+0000 7fe9867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.693+0000 7fe9867fc640 10 monclient(hunting): dump: 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:02.689 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.693+0000 7fe9867fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:08:02.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.694+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2634977560 0 0) 0x7fe98802e700 con 0x7fe988030660 2024-03-27T03:08:02.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.694+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 >> v1:172.21.15.153:6790/0 conn(0x7fe9880389e0 legacy=0x7fe98802faf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:08:02.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 >> v1:172.21.15.153:6789/0 conn(0x7fe98816d3e0 legacy=0x7fe9880c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:02.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe9867fc640 10 monclient: _finish_hunting 0 2024-03-27T03:08:02.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe9867fc640 1 monclient: found mon.b 2024-03-27T03:08:02.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe9867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:02.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe9867fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:02.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98800d7b0 con 0x7fe988030660 2024-03-27T03:08:02.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe9867fc640 10 monclient: _finish_auth 0 2024-03-27T03:08:02.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe9867fc640 10 monclient: _check_auth_tickets 2024-03-27T03:08:02.693 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe9867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:02.694 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe996bb4640 5 monclient: authenticate success, global_id 4327 2024-03-27T03:08:02.694 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe996bb4640 10 monclient: _renew_subs 2024-03-27T03:08:02.694 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe996bb4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:02.694 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe996bb4640 1 -- 172.21.15.153:0/326397142 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe98802d150 con 0x7fe988030660 2024-03-27T03:08:02.694 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe996bb4640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:08:02.694 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe996bb4640 10 monclient: _renew_subs 2024-03-27T03:08:02.694 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe996bb4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:02.694 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe996bb4640 1 -- 172.21.15.153:0/326397142 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9880c4950 con 0x7fe988030660 2024-03-27T03:08:02.694 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.695+0000 7fe996bb4640 1 librados: init done 2024-03-27T03:08:02.694 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.696+0000 7fe9867fc640 10 client.4327.objecter ms_handle_connect 0x7fe98816d3e0 2024-03-27T03:08:02.694 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.696+0000 7fe9867fc640 10 client.4327.objecter resend_mon_ops 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.696+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe98c0016f0 con 0x7fe988030660 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.696+0000 7fe9867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.696+0000 7fe987fff640 4 set_mon_vals no callback set 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.697+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe98c004c30 con 0x7fe988030660 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.697+0000 7fe9867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.697+0000 7fe9867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.697+0000 7fe9867fc640 10 monclient: dump: 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:02.695 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:02.696 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:02.696 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:02.696 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.697+0000 7fe996bb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:02.696 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.697+0000 7fe996bb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:02.696 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.697+0000 7fe996bb4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:02.696 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.697+0000 7fe996bb4640 1 -- 172.21.15.153:0/326397142 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe978005080 con 0x7fe988030660 2024-03-27T03:08:02.696 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.699+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fe98c01ac00 con 0x7fe988030660 2024-03-27T03:08:02.696 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.700+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fe98c084a90 con 0x7fe988030660 2024-03-27T03:08:02.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.700+0000 7fe9867fc640 10 client.4327.objecter ms_dispatch 0x7fe990000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:08:02.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.700+0000 7fe9867fc640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:08:02.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.700+0000 7fe9867fc640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:08:02.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.700+0000 7fe9867fc640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-27T03:08:02.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.700+0000 7fe9953b1640 10 monclient: get_auth_request con 0x7fe97406db60 auth_method 0 2024-03-27T03:08:02.697 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.701+0000 7fe9867fc640 10 client.4327.objecter ms_handle_connect 0x7fe97406db60 2024-03-27T03:08:02.701 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.706+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe98c0501a0 con 0x7fe988030660 2024-03-27T03:08:02.701 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.706+0000 7fe9867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:02.701 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:02.706+0000 7fe9867fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:08:03.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.035+0000 7fe996bb4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:03.029 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.035+0000 7fe996bb4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:03.029 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.035+0000 7fe996bb4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:03.029 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.035+0000 7fe996bb4640 1 -- 172.21.15.153:0/326397142 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe978005330 con 0x7fe988030660 2024-03-27T03:08:03.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.038+0000 7fe9867fc640 1 -- 172.21.15.153:0/326397142 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fe98c0522a0 con 0x7fe988030660 2024-03-27T03:08:03.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.038+0000 7fe9867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:03.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.038+0000 7fe9867fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:08:03.032 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:08:03.032 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:08:03.035 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.042+0000 7fe996bb4640 10 librados: watch_flush enter 2024-03-27T03:08:03.035 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.043+0000 7fe996bb4640 10 librados: watch_flush exit 2024-03-27T03:08:03.035 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.043+0000 7fe996bb4640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-27T03:08:03.035 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.043+0000 7fe996bb4640 10 client.4327.objecter successfully canceled tick 2024-03-27T03:08:03.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.044+0000 7fe996bb4640 1 -- 172.21.15.153:0/326397142 >> v1:172.21.15.153:6816/2637937781 conn(0x7fe97406db60 legacy=0x7fe974070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:03.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.045+0000 7fe996bb4640 10 monclient: shutdown 2024-03-27T03:08:03.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.045+0000 7fe996bb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:03.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.045+0000 7fe996bb4640 1 -- 172.21.15.153:0/326397142 >> v1:172.21.15.196:6789/0 conn(0x7fe988030660 legacy=0x7fe98816ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:03.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.045+0000 7fe996bb4640 1 -- 172.21.15.153:0/326397142 shutdown_connections 2024-03-27T03:08:03.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.046+0000 7fe996bb4640 1 -- 172.21.15.153:0/326397142 >> 172.21.15.153:0/326397142 conn(0x7fe988005c00 msgr2=0x7fe988034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:03.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.046+0000 7fe996bb4640 1 -- 172.21.15.153:0/326397142 shutdown_connections 2024-03-27T03:08:03.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.046+0000 7fe996bb4640 1 -- 172.21.15.153:0/326397142 wait complete. 2024-03-27T03:08:03.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:03.046+0000 7fe996bb4640 1 librados: shutdown 2024-03-27T03:08:03.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:08:09.868 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:08:10.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.023+0000 7f2712858640 10 monclient: get_monmap_and_config 2024-03-27T03:08:10.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.023+0000 7f2712858640 10 monclient: build_initial_monmap 2024-03-27T03:08:10.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.023+0000 7f2712858640 10 monclient: monmap: 2024-03-27T03:08:10.017 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:10.017 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:10.017 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:08:10.024876+0000 2024-03-27T03:08:10.017 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:08:10.024876+0000 2024-03-27T03:08:10.017 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:10.017 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:10.017 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:10.017 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:08:10.017 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:08:10.017 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:10.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.024+0000 7f2712858640 1 Processor -- start 2024-03-27T03:08:10.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.025+0000 7f2712858640 1 -- start start 2024-03-27T03:08:10.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.025+0000 7f2712858640 10 monclient: init 2024-03-27T03:08:10.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.026+0000 7f2712858640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:10.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.026+0000 7f2712858640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:08:10.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.026+0000 7f2712858640 10 monclient(hunting): picked mon.noname-a con 0x7f270c0b2790 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:10.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.026+0000 7f2712858640 10 monclient(hunting): picked mon.noname-b con 0x7f270c143c80 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:10.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.026+0000 7f2712858640 10 monclient(hunting): picked mon.noname-c con 0x7f270c146e30 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:10.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.026+0000 7f2712858640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c146290 con 0x7f270c0b2790 2024-03-27T03:08:10.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.026+0000 7f2712858640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c0b3970 con 0x7f270c146e30 2024-03-27T03:08:10.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.026+0000 7f2712858640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c06ee80 con 0x7f270c143c80 2024-03-27T03:08:10.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.026+0000 7f2712858640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:10.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.026+0000 7f2712858640 10 monclient(hunting): authenticate will time out at 1450.332446s 2024-03-27T03:08:10.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.030+0000 7f2711856640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f270c0b2790 0x7f270c143570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44423/0 (socket says 172.21.15.153:44423) 2024-03-27T03:08:10.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.030+0000 7f2711856640 1 -- 172.21.15.153:0/3067421359 learned_addr learned my addr 172.21.15.153:0/3067421359 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:10.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.031+0000 7f2710854640 1 -- 172.21.15.153:0/3067421359 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269125668 0 0) 0x7f270c146290 con 0x7f270c0b2790 2024-03-27T03:08:10.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.031+0000 7f2710854640 10 monclient(hunting): _negotiate 2024-03-27T03:08:10.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.031+0000 7f2710854640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:10.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.031+0000 7f2710854640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:10.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.031+0000 7f2710854640 10 monclient(hunting): my global_id is 4317 2024-03-27T03:08:10.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.031+0000 7f2710854640 1 -- 172.21.15.153:0/3067421359 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f8003520 con 0x7f270c0b2790 2024-03-27T03:08:10.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.031+0000 7f2710854640 1 -- 172.21.15.153:0/3067421359 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1717001105 0 0) 0x7f26f8003520 con 0x7f270c0b2790 2024-03-27T03:08:10.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.031+0000 7f2710854640 1 -- 172.21.15.153:0/3067421359 >> v1:172.21.15.153:6790/0 conn(0x7f270c143c80 legacy=0x7f270c146720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:08:10.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 1 -- 172.21.15.153:0/3067421359 >> v1:172.21.15.196:6789/0 conn(0x7f270c146e30 legacy=0x7f270c151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:10.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 10 monclient: _finish_hunting 0 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 1 monclient: found mon.noname-a 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 1 -- 172.21.15.153:0/3067421359 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c06f160 con 0x7f270c0b2790 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 10 monclient: _finish_auth 0 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 10 monclient: _check_auth_tickets 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2712858640 5 monclient: authenticate success, global_id 4317 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2712858640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 1 -- 172.21.15.153:0/3067421359 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2708001690 con 0x7f270c0b2790 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 10 monclient: dump: 2024-03-27T03:08:10.026 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:10.027 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:10.027 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:10.027 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:10.027 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:10.027 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:10.027 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:10.027 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:10.027 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:10.027 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:10.027 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2712858640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:08:10.027 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 1 -- 172.21.15.153:0/3067421359 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2708001920 con 0x7f270c0b2790 2024-03-27T03:08:10.027 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2710854640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:10.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2712858640 10 monclient: get_monmap_and_config success 2024-03-27T03:08:10.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.032+0000 7f2712858640 4 set_mon_vals no callback set 2024-03-27T03:08:10.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.033+0000 7f2712858640 10 monclient: shutdown 2024-03-27T03:08:10.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.033+0000 7f2712858640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:10.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.033+0000 7f2712858640 1 -- 172.21.15.153:0/3067421359 >> v1:172.21.15.153:6789/0 conn(0x7f270c0b2790 legacy=0x7f270c143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.033+0000 7f2712858640 1 -- 172.21.15.153:0/3067421359 shutdown_connections 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.033+0000 7f2712858640 1 -- 172.21.15.153:0/3067421359 >> 172.21.15.153:0/3067421359 conn(0x7f270c06d070 msgr2=0x7f270c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.033+0000 7f2712858640 1 -- 172.21.15.153:0/3067421359 shutdown_connections 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.033+0000 7f2712858640 1 -- 172.21.15.153:0/3067421359 wait complete. 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.034+0000 7f2712858640 10 monclient: build_initial_monmap 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.034+0000 7f2712858640 10 monclient: monmap: 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:08:10.029 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:08:10.030 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:10.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.035+0000 7f2712858640 1 librados: starting msgr at 2024-03-27T03:08:10.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.035+0000 7f2712858640 1 librados: starting objecter 2024-03-27T03:08:10.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.035+0000 7f2712858640 1 Processor -- start 2024-03-27T03:08:10.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.035+0000 7f2712858640 1 -- start start 2024-03-27T03:08:10.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.035+0000 7f2712858640 1 librados: setting wanted keys 2024-03-27T03:08:10.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.035+0000 7f2712858640 1 librados: calling monclient init 2024-03-27T03:08:10.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.035+0000 7f2712858640 10 monclient: init 2024-03-27T03:08:10.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.036+0000 7f2712858640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:10.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.036+0000 7f2712858640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:08:10.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.036+0000 7f2712858640 10 monclient(hunting): picked mon.noname-a con 0x7f270c0b2790 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:10.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.036+0000 7f2712858640 10 monclient(hunting): picked mon.noname-b con 0x7f270c143c80 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:10.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.036+0000 7f2712858640 10 monclient(hunting): picked mon.noname-c con 0x7f270c146e30 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:10.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.036+0000 7f2712858640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c148f30 con 0x7f270c0b2790 2024-03-27T03:08:10.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.036+0000 7f2712858640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c1479a0 con 0x7f270c143c80 2024-03-27T03:08:10.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.036+0000 7f2712858640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c13b540 con 0x7f270c146e30 2024-03-27T03:08:10.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.036+0000 7f2712858640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:10.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.036+0000 7f2712858640 10 monclient(hunting): authenticate will time out at 1450.341946s 2024-03-27T03:08:10.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.038+0000 7f2711055640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f270c143c80 0x7f270c1db030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:49720/0 (socket says 172.21.15.153:49720) 2024-03-27T03:08:10.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.038+0000 7f2711055640 1 -- 172.21.15.153:0/2593854915 learned_addr learned my addr 172.21.15.153:0/2593854915 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:10.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.038+0000 7f27027fc640 10 client.?.objecter ms_handle_connect 0x7f270c143c80 2024-03-27T03:08:10.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.038+0000 7f27027fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:10.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 client.?.objecter ms_handle_connect 0x7f270c146e30 2024-03-27T03:08:10.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:10.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408665090 0 0) 0x7f270c13b540 con 0x7f270c146e30 2024-03-27T03:08:10.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient(hunting): _negotiate 2024-03-27T03:08:10.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:10.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:10.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient(hunting): my global_id is 4337 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f0004eb0 con 0x7f270c146e30 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214361338 0 0) 0x7f270c1479a0 con 0x7f270c143c80 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient(hunting): _negotiate 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient(hunting): my global_id is 4330 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f0006230 con 0x7f270c143c80 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 918834582 0 0) 0x7f26f0004eb0 con 0x7f270c146e30 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f270c1479a0 con 0x7f270c146e30 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f26fc001810 con 0x7f270c146e30 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient(hunting): dump: 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:10.033 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:10.034 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:10.034 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:10.034 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:10.034 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:10.034 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:10.034 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:10.034 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:10.034 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:10.034 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:08:10.035 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2744977054 0 0) 0x7f270c1479a0 con 0x7f270c146e30 2024-03-27T03:08:10.035 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 >> v1:172.21.15.196:6789/0 conn(0x7f270c143c80 legacy=0x7f270c1db030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:10.035 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 >> v1:172.21.15.153:6789/0 conn(0x7f270c0b2790 legacy=0x7f270c13ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:10.035 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient: _finish_hunting 0 2024-03-27T03:08:10.035 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 1 monclient: found mon.c 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c13b6d0 con 0x7f270c146e30 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient: _finish_auth 0 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 10 monclient: _check_auth_tickets 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f27027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f2712858640 5 monclient: authenticate success, global_id 4337 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f2712858640 10 monclient: _renew_subs 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f2712858640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.039+0000 7f2712858640 1 -- 172.21.15.153:0/2593854915 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270c0b7550 con 0x7f270c146e30 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.040+0000 7f2712858640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.040+0000 7f2712858640 10 monclient: _renew_subs 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.040+0000 7f2712858640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.040+0000 7f2712858640 1 -- 172.21.15.153:0/2593854915 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f270c1475a0 con 0x7f270c146e30 2024-03-27T03:08:10.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.040+0000 7f2712858640 1 librados: init done 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.041+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26fc001cc0 con 0x7f270c146e30 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.041+0000 7f27027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.041+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f26fc001810 con 0x7f270c146e30 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.041+0000 7f27027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.041+0000 7f27027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.041+0000 7f27027fc640 10 monclient: dump: 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:10.037 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:10.038 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:10.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.042+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f26fc0194e0 con 0x7f270c146e30 2024-03-27T03:08:10.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.042+0000 7f2703fff640 4 set_mon_vals no callback set 2024-03-27T03:08:10.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.043+0000 7f2711856640 10 monclient: get_auth_request con 0x7f26f006e290 auth_method 0 2024-03-27T03:08:10.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.043+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f26fc083220 con 0x7f270c146e30 2024-03-27T03:08:10.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.043+0000 7f27027fc640 10 client.4337.objecter ms_dispatch 0x7f270c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:08:10.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.043+0000 7f27027fc640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:08:10.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.043+0000 7f27027fc640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:08:10.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.043+0000 7f27027fc640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-27T03:08:10.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.043+0000 7f27027fc640 10 client.4337.objecter ms_handle_connect 0x7f26f006e290 2024-03-27T03:08:10.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.044+0000 7f2712858640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:10.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.044+0000 7f2712858640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:10.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.044+0000 7f2712858640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:10.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.044+0000 7f2712858640 1 -- 172.21.15.153:0/2593854915 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f4005080 con 0x7f270c146e30 2024-03-27T03:08:10.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.052+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26fc04eb60 con 0x7f270c146e30 2024-03-27T03:08:10.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.052+0000 7f27027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:10.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.052+0000 7f27027fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:08:10.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.384+0000 7f2712858640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:10.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.384+0000 7f2712858640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:10.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.384+0000 7f2712858640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:10.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.384+0000 7f2712858640 1 -- 172.21.15.153:0/2593854915 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26f4005330 con 0x7f270c146e30 2024-03-27T03:08:10.380 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:08:10.380 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:08:10.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.386+0000 7f27027fc640 1 -- 172.21.15.153:0/2593854915 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f26fc050a90 con 0x7f270c146e30 2024-03-27T03:08:10.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.386+0000 7f27027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:10.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.386+0000 7f27027fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:08:10.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.390+0000 7f2712858640 10 librados: watch_flush enter 2024-03-27T03:08:10.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.390+0000 7f2712858640 10 librados: watch_flush exit 2024-03-27T03:08:10.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.390+0000 7f2712858640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-27T03:08:10.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.390+0000 7f2712858640 10 client.4337.objecter successfully canceled tick 2024-03-27T03:08:10.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.390+0000 7f2712858640 1 -- 172.21.15.153:0/2593854915 >> v1:172.21.15.153:6816/2637937781 conn(0x7f26f006e290 legacy=0x7f26f0070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:10.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.390+0000 7f2712858640 10 monclient: shutdown 2024-03-27T03:08:10.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.390+0000 7f2712858640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:10.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.390+0000 7f2712858640 1 -- 172.21.15.153:0/2593854915 >> v1:172.21.15.153:6790/0 conn(0x7f270c146e30 legacy=0x7f270c1e68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:10.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.390+0000 7f2712858640 1 -- 172.21.15.153:0/2593854915 shutdown_connections 2024-03-27T03:08:10.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.390+0000 7f2712858640 1 -- 172.21.15.153:0/2593854915 >> 172.21.15.153:0/2593854915 conn(0x7f270c06d070 msgr2=0x7f270c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:10.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.391+0000 7f2712858640 1 -- 172.21.15.153:0/2593854915 shutdown_connections 2024-03-27T03:08:10.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.391+0000 7f2712858640 1 -- 172.21.15.153:0/2593854915 wait complete. 2024-03-27T03:08:10.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:10.393+0000 7f2712858640 1 librados: shutdown 2024-03-27T03:08:10.402 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:08:17.208 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:08:17.368 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.375+0000 7f8f4ead3640 10 monclient: get_monmap_and_config 2024-03-27T03:08:17.368 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.375+0000 7f8f4ead3640 10 monclient: build_initial_monmap 2024-03-27T03:08:17.368 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.375+0000 7f8f4ead3640 10 monclient: monmap: 2024-03-27T03:08:17.368 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:17.368 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:17.368 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:08:17.376067+0000 2024-03-27T03:08:17.368 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:08:17.376067+0000 2024-03-27T03:08:17.369 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:17.369 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:17.369 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:17.369 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:08:17.369 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:08:17.369 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:17.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.376+0000 7f8f4ead3640 1 Processor -- start 2024-03-27T03:08:17.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.376+0000 7f8f4ead3640 1 -- start start 2024-03-27T03:08:17.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.376+0000 7f8f4ead3640 10 monclient: init 2024-03-27T03:08:17.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.377+0000 7f8f4ead3640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:17.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.377+0000 7f8f4ead3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:08:17.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.377+0000 7f8f4ead3640 10 monclient(hunting): picked mon.noname-a con 0x7f8f480b0770 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:17.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.377+0000 7f8f4ead3640 10 monclient(hunting): picked mon.noname-b con 0x7f8f4814a700 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:17.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.377+0000 7f8f4ead3640 10 monclient(hunting): picked mon.noname-c con 0x7f8f480ac5a0 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:17.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.377+0000 7f8f4ead3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f480aba70 con 0x7f8f480b0770 2024-03-27T03:08:17.372 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.377+0000 7f8f4ead3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f480aecb0 con 0x7f8f480ac5a0 2024-03-27T03:08:17.372 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.377+0000 7f8f4ead3640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f480af150 con 0x7f8f4814a700 2024-03-27T03:08:17.372 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.377+0000 7f8f4ead3640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:17.372 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.377+0000 7f8f4ead3640 10 monclient(hunting): authenticate will time out at 1457.682805s 2024-03-27T03:08:17.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.384+0000 7f8f47fff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f8f4814a700 0x7f8f480abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:60847/0 (socket says 172.21.15.153:60847) 2024-03-27T03:08:17.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.384+0000 7f8f47fff640 1 -- 172.21.15.153:0/3733477905 learned_addr learned my addr 172.21.15.153:0/3733477905 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:17.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673706338 0 0) 0x7f8f480aba70 con 0x7f8f480b0770 2024-03-27T03:08:17.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient(hunting): _negotiate 2024-03-27T03:08:17.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:17.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:17.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient(hunting): my global_id is 4320 2024-03-27T03:08:17.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f38003520 con 0x7f8f480b0770 2024-03-27T03:08:17.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277607717 0 0) 0x7f8f480af150 con 0x7f8f4814a700 2024-03-27T03:08:17.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient(hunting): _negotiate 2024-03-27T03:08:17.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:17.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:17.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient(hunting): my global_id is 4340 2024-03-27T03:08:17.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f480aba70 con 0x7f8f4814a700 2024-03-27T03:08:17.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011780101 0 0) 0x7f8f480aecb0 con 0x7f8f480ac5a0 2024-03-27T03:08:17.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient(hunting): _negotiate 2024-03-27T03:08:17.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:17.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:17.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient(hunting): my global_id is 4333 2024-03-27T03:08:17.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f480af150 con 0x7f8f480ac5a0 2024-03-27T03:08:17.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 10887931 0 0) 0x7f8f480aba70 con 0x7f8f4814a700 2024-03-27T03:08:17.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 >> v1:172.21.15.196:6789/0 conn(0x7f8f480ac5a0 legacy=0x7f8f48151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:17.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 >> v1:172.21.15.153:6789/0 conn(0x7f8f480b0770 legacy=0x7f8f48149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:17.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient: _finish_hunting 0 2024-03-27T03:08:17.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 monclient: found mon.noname-b 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f480ac9d0 con 0x7f8f4814a700 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient: _finish_auth 0 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient: _check_auth_tickets 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8f3c001520 con 0x7f8f4814a700 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient: dump: 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:17.389 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f4ead3640 5 monclient: authenticate success, global_id 4340 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f4ead3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f3c001680 con 0x7f8f4814a700 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 1 -- 172.21.15.153:0/3733477905 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8f3c001a90 con 0x7f8f4814a700 2024-03-27T03:08:17.390 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f477fe640 10 monclient: dump: 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:17.391 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f4ead3640 10 monclient: get_monmap_and_config success 2024-03-27T03:08:17.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.386+0000 7f8f4ead3640 4 set_mon_vals no callback set 2024-03-27T03:08:17.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.387+0000 7f8f4ead3640 10 monclient: shutdown 2024-03-27T03:08:17.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.387+0000 7f8f4ead3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:17.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.387+0000 7f8f4ead3640 1 -- 172.21.15.153:0/3733477905 >> v1:172.21.15.153:6790/0 conn(0x7f8f4814a700 legacy=0x7f8f480abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:17.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.393+0000 7f8f4ead3640 1 -- 172.21.15.153:0/3733477905 shutdown_connections 2024-03-27T03:08:17.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.393+0000 7f8f4ead3640 1 -- 172.21.15.153:0/3733477905 >> 172.21.15.153:0/3733477905 conn(0x7f8f4806b660 msgr2=0x7f8f4806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:17.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.393+0000 7f8f4ead3640 1 -- 172.21.15.153:0/3733477905 shutdown_connections 2024-03-27T03:08:17.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.393+0000 7f8f4ead3640 1 -- 172.21.15.153:0/3733477905 wait complete. 2024-03-27T03:08:17.393 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.394+0000 7f8f4ead3640 10 monclient: build_initial_monmap 2024-03-27T03:08:17.393 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.394+0000 7f8f4ead3640 10 monclient: monmap: 2024-03-27T03:08:17.393 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:17.393 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:08:17.393 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:08:17.393 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:08:17.393 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:17.393 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:17.394 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:17.394 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:08:17.394 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:08:17.394 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:17.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.395+0000 7f8f4ead3640 1 librados: starting msgr at 2024-03-27T03:08:17.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.395+0000 7f8f4ead3640 1 librados: starting objecter 2024-03-27T03:08:17.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.395+0000 7f8f4ead3640 1 Processor -- start 2024-03-27T03:08:17.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.397+0000 7f8f4ead3640 1 -- start start 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.397+0000 7f8f4ead3640 1 librados: setting wanted keys 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.397+0000 7f8f4ead3640 1 librados: calling monclient init 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.397+0000 7f8f4ead3640 10 monclient: init 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.398+0000 7f8f4ead3640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.398+0000 7f8f4ead3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.398+0000 7f8f4ead3640 10 monclient(hunting): picked mon.noname-a con 0x7f8f480ac5a0 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.398+0000 7f8f4ead3640 10 monclient(hunting): picked mon.noname-c con 0x7f8f480b0770 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.398+0000 7f8f4ead3640 10 monclient(hunting): picked mon.noname-b con 0x7f8f4814a700 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.398+0000 7f8f4ead3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f48123140 con 0x7f8f480ac5a0 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.398+0000 7f8f4ead3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f48121bc0 con 0x7f8f4814a700 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.398+0000 7f8f4ead3640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f48122d40 con 0x7f8f480b0770 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.398+0000 7f8f4ead3640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:17.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.398+0000 7f8f4ead3640 10 monclient(hunting): authenticate will time out at 1457.704515s 2024-03-27T03:08:17.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.403+0000 7f8f4c86f640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8f480ac5a0 0x7f8f48122630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:46605/0 (socket says 172.21.15.153:46605) 2024-03-27T03:08:17.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.403+0000 7f8f4c86f640 1 -- 172.21.15.153:0/2252166372 learned_addr learned my addr 172.21.15.153:0/2252166372 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:17.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.403+0000 7f8f457fa640 10 client.?.objecter ms_handle_connect 0x7f8f480ac5a0 2024-03-27T03:08:17.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.403+0000 7f8f457fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:17.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.403+0000 7f8f457fa640 10 client.?.objecter ms_handle_connect 0x7f8f480b0770 2024-03-27T03:08:17.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.403+0000 7f8f457fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:17.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.403+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383429921 0 0) 0x7f8f48122d40 con 0x7f8f480b0770 2024-03-27T03:08:17.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.403+0000 7f8f457fa640 10 monclient(hunting): _negotiate 2024-03-27T03:08:17.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.404+0000 7f8f457fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:17.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.404+0000 7f8f457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:17.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.404+0000 7f8f457fa640 10 monclient(hunting): my global_id is 4343 2024-03-27T03:08:17.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.404+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f2c004eb0 con 0x7f8f480b0770 2024-03-27T03:08:17.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011888789 0 0) 0x7f8f48123140 con 0x7f8f480ac5a0 2024-03-27T03:08:17.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 10 monclient(hunting): _negotiate 2024-03-27T03:08:17.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:17.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:17.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 10 monclient(hunting): my global_id is 4323 2024-03-27T03:08:17.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f48122d40 con 0x7f8f480ac5a0 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2172114068 0 0) 0x7f8f2c004eb0 con 0x7f8f480b0770 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f48123140 con 0x7f8f480b0770 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8f3c001340 con 0x7f8f480b0770 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 10 monclient(hunting): dump: 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:17.404 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:17.405 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:17.405 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:17.405 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:17.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.406+0000 7f8f457fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:08:17.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2225829476 0 0) 0x7f8f48123140 con 0x7f8f480b0770 2024-03-27T03:08:17.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 >> v1:172.21.15.196:6789/0 conn(0x7f8f4814a700 legacy=0x7f8f481e4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:17.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 >> v1:172.21.15.153:6789/0 conn(0x7f8f480ac5a0 legacy=0x7f8f48122630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:17.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f457fa640 10 monclient: _finish_hunting 0 2024-03-27T03:08:17.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f457fa640 1 monclient: found mon.c 2024-03-27T03:08:17.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:17.405 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f457fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f48148180 con 0x7f8f480b0770 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f457fa640 10 monclient: _finish_auth 0 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f457fa640 10 monclient: _check_auth_tickets 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f4ead3640 5 monclient: authenticate success, global_id 4343 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f4ead3640 10 monclient: _renew_subs 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f4ead3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f4ead3640 1 -- 172.21.15.153:0/2252166372 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f48148450 con 0x7f8f480b0770 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f4ead3640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f4ead3640 10 monclient: _renew_subs 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f4ead3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f4ead3640 1 -- 172.21.15.153:0/2252166372 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f481efa30 con 0x7f8f480b0770 2024-03-27T03:08:17.406 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.407+0000 7f8f4ead3640 1 librados: init done 2024-03-27T03:08:17.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.408+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f3c001500 con 0x7f8f480b0770 2024-03-27T03:08:17.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.408+0000 7f8f457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:17.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.408+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8f3c0010e0 con 0x7f8f480b0770 2024-03-27T03:08:17.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.408+0000 7f8f457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:17.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.408+0000 7f8f457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:08:17.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.408+0000 7f8f457fa640 10 monclient: dump: 2024-03-27T03:08:17.407 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:17.407 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:17.407 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:17.407 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:17.407 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:17.407 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:17.408 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:17.408 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:17.408 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:17.408 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:17.408 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.408+0000 7f8f46ffd640 4 set_mon_vals no callback set 2024-03-27T03:08:17.409 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.410+0000 7f8f4ead3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:17.409 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.410+0000 7f8f4ead3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:17.410 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.410+0000 7f8f4ead3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:17.410 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.410+0000 7f8f4ead3640 1 -- 172.21.15.153:0/2252166372 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f28005080 con 0x7f8f480b0770 2024-03-27T03:08:17.410 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.412+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f8f3c01aaa0 con 0x7f8f480b0770 2024-03-27T03:08:17.410 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.413+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f8f3c084a20 con 0x7f8f480b0770 2024-03-27T03:08:17.410 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.413+0000 7f8f457fa640 10 client.4343.objecter ms_dispatch 0x7f8f48000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:08:17.410 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.413+0000 7f8f457fa640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:08:17.410 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.413+0000 7f8f457fa640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:08:17.410 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.413+0000 7f8f457fa640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-27T03:08:17.410 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.414+0000 7f8f4c86f640 10 monclient: get_auth_request con 0x7f8f2c06ddd0 auth_method 0 2024-03-27T03:08:17.410 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.414+0000 7f8f457fa640 10 client.4343.objecter ms_handle_connect 0x7f8f2c06ddd0 2024-03-27T03:08:17.417 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.424+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f3c050130 con 0x7f8f480b0770 2024-03-27T03:08:17.417 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.425+0000 7f8f457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:17.417 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.425+0000 7f8f457fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:08:17.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.750+0000 7f8f4ead3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:17.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.750+0000 7f8f4ead3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:17.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.750+0000 7f8f4ead3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:17.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.750+0000 7f8f4ead3640 1 -- 172.21.15.153:0/2252166372 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f28005330 con 0x7f8f480b0770 2024-03-27T03:08:17.744 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.751+0000 7f8f457fa640 1 -- 172.21.15.153:0/2252166372 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f8f3c052230 con 0x7f8f480b0770 2024-03-27T03:08:17.744 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.751+0000 7f8f457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:17.744 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.751+0000 7f8f457fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:08:17.744 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:08:17.745 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:08:17.751 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.758+0000 7f8f4ead3640 10 librados: watch_flush enter 2024-03-27T03:08:17.751 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.758+0000 7f8f4ead3640 10 librados: watch_flush exit 2024-03-27T03:08:17.751 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.758+0000 7f8f4ead3640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-27T03:08:17.751 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.758+0000 7f8f4ead3640 10 client.4343.objecter successfully canceled tick 2024-03-27T03:08:17.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.763+0000 7f8f4ead3640 1 -- 172.21.15.153:0/2252166372 >> v1:172.21.15.153:6816/2637937781 conn(0x7f8f2c06ddd0 legacy=0x7f8f2c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:17.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.763+0000 7f8f4ead3640 10 monclient: shutdown 2024-03-27T03:08:17.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.763+0000 7f8f4ead3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:17.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.763+0000 7f8f4ead3640 1 -- 172.21.15.153:0/2252166372 >> v1:172.21.15.153:6790/0 conn(0x7f8f480b0770 legacy=0x7f8f481e4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:17.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.763+0000 7f8f4ead3640 1 -- 172.21.15.153:0/2252166372 shutdown_connections 2024-03-27T03:08:17.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.763+0000 7f8f4ead3640 1 -- 172.21.15.153:0/2252166372 >> 172.21.15.153:0/2252166372 conn(0x7f8f4806b660 msgr2=0x7f8f4806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:17.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.763+0000 7f8f4ead3640 1 -- 172.21.15.153:0/2252166372 shutdown_connections 2024-03-27T03:08:17.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.763+0000 7f8f4ead3640 1 -- 172.21.15.153:0/2252166372 wait complete. 2024-03-27T03:08:17.757 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:17.764+0000 7f8f4ead3640 1 librados: shutdown 2024-03-27T03:08:17.774 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:08:24.581 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:08:24.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.738+0000 7f0389401640 10 monclient: get_monmap_and_config 2024-03-27T03:08:24.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.738+0000 7f0389401640 10 monclient: build_initial_monmap 2024-03-27T03:08:24.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.738+0000 7f0389401640 10 monclient: monmap: 2024-03-27T03:08:24.733 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:24.733 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:24.733 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:08:24.739784+0000 2024-03-27T03:08:24.733 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:08:24.739784+0000 2024-03-27T03:08:24.733 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:24.733 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:24.733 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:24.733 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:08:24.733 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:08:24.733 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:24.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.741+0000 7f0389401640 1 Processor -- start 2024-03-27T03:08:24.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.741+0000 7f0389401640 1 -- start start 2024-03-27T03:08:24.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.741+0000 7f0389401640 10 monclient: init 2024-03-27T03:08:24.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.742+0000 7f0389401640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:24.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.742+0000 7f0389401640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:08:24.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.742+0000 7f0389401640 10 monclient(hunting): picked mon.noname-a con 0x7f03840a9ac0 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:24.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.742+0000 7f0389401640 10 monclient(hunting): picked mon.noname-c con 0x7f038413ee90 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:24.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.742+0000 7f0389401640 10 monclient(hunting): picked mon.noname-b con 0x7f03840af900 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:24.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.742+0000 7f0389401640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03840afd10 con 0x7f03840a9ac0 2024-03-27T03:08:24.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.742+0000 7f0389401640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03840af6f0 con 0x7f038413ee90 2024-03-27T03:08:24.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.742+0000 7f0389401640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03840aac80 con 0x7f03840af900 2024-03-27T03:08:24.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.742+0000 7f0389401640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:24.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.742+0000 7f0389401640 10 monclient(hunting): authenticate will time out at 1465.048059s 2024-03-27T03:08:24.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.745+0000 7f03837fe640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f03840af900 0x7f0384149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:35239/0 (socket says 172.21.15.153:35239) 2024-03-27T03:08:24.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.745+0000 7f03837fe640 1 -- 172.21.15.153:0/2708935488 learned_addr learned my addr 172.21.15.153:0/2708935488 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:24.740 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.746+0000 7f0381ffb640 1 -- 172.21.15.153:0/2708935488 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597142892 0 0) 0x7f03840aac80 con 0x7f03840af900 2024-03-27T03:08:24.740 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.746+0000 7f0381ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:08:24.740 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.746+0000 7f0381ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:24.740 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.746+0000 7f0381ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:24.740 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.746+0000 7f0381ffb640 10 monclient(hunting): my global_id is 4346 2024-03-27T03:08:24.740 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.746+0000 7f0381ffb640 1 -- 172.21.15.153:0/2708935488 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f036c003520 con 0x7f03840af900 2024-03-27T03:08:24.740 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.747+0000 7f0381ffb640 1 -- 172.21.15.153:0/2708935488 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1950701326 0 0) 0x7f036c003520 con 0x7f03840af900 2024-03-27T03:08:24.740 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.747+0000 7f0381ffb640 1 -- 172.21.15.153:0/2708935488 >> v1:172.21.15.196:6789/0 conn(0x7f038413ee90 legacy=0x7f038413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:24.740 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.747+0000 7f0381ffb640 1 -- 172.21.15.153:0/2708935488 >> v1:172.21.15.153:6789/0 conn(0x7f03840a9ac0 legacy=0x7f03840adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:08:24.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.748+0000 7f0381ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:08:24.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 1 monclient: found mon.noname-b 2024-03-27T03:08:24.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:24.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:08:24.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 1 -- 172.21.15.153:0/2708935488 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03840aaf60 con 0x7f03840af900 2024-03-27T03:08:24.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 10 monclient: _finish_auth 0 2024-03-27T03:08:24.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:08:24.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:24.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 1 -- 172.21.15.153:0/2708935488 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0374001a10 con 0x7f03840af900 2024-03-27T03:08:24.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:24.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:08:24.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 10 monclient: dump: 2024-03-27T03:08:24.743 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:24.743 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:24.743 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:24.743 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:24.743 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:24.743 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:24.743 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:24.743 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:24.743 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:24.743 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:24.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 1 -- 172.21.15.153:0/2708935488 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0374001ca0 con 0x7f03840af900 2024-03-27T03:08:24.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 1 -- 172.21.15.153:0/2708935488 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0374002030 con 0x7f03840af900 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.749+0000 7f0381ffb640 10 monclient: dump: 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:24.744 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:24.745 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.751+0000 7f0389401640 5 monclient: authenticate success, global_id 4346 2024-03-27T03:08:24.745 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.751+0000 7f0389401640 10 monclient: get_monmap_and_config success 2024-03-27T03:08:24.745 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.751+0000 7f0389401640 4 set_mon_vals no callback set 2024-03-27T03:08:24.745 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.752+0000 7f0389401640 10 monclient: shutdown 2024-03-27T03:08:24.745 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.752+0000 7f0389401640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:24.745 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.752+0000 7f0389401640 1 -- 172.21.15.153:0/2708935488 >> v1:172.21.15.153:6790/0 conn(0x7f03840af900 legacy=0x7f0384149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:24.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.753+0000 7f0389401640 1 -- 172.21.15.153:0/2708935488 shutdown_connections 2024-03-27T03:08:24.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.753+0000 7f0389401640 1 -- 172.21.15.153:0/2708935488 >> 172.21.15.153:0/2708935488 conn(0x7f03840b35e0 msgr2=0x7f03840b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:24.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.753+0000 7f0389401640 1 -- 172.21.15.153:0/2708935488 shutdown_connections 2024-03-27T03:08:24.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.754+0000 7f0389401640 1 -- 172.21.15.153:0/2708935488 wait complete. 2024-03-27T03:08:24.749 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.755+0000 7f0389401640 10 monclient: build_initial_monmap 2024-03-27T03:08:24.749 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.755+0000 7f0389401640 10 monclient: monmap: 2024-03-27T03:08:24.749 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:24.749 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:08:24.749 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:08:24.749 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:08:24.749 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:24.749 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:24.749 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:24.749 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:08:24.750 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:08:24.750 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:24.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.756+0000 7f0389401640 1 librados: starting msgr at 2024-03-27T03:08:24.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.756+0000 7f0389401640 1 librados: starting objecter 2024-03-27T03:08:24.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.756+0000 7f0389401640 1 Processor -- start 2024-03-27T03:08:24.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.759+0000 7f0389401640 1 -- start start 2024-03-27T03:08:24.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.759+0000 7f0389401640 1 librados: setting wanted keys 2024-03-27T03:08:24.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.759+0000 7f0389401640 1 librados: calling monclient init 2024-03-27T03:08:24.753 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.759+0000 7f0389401640 10 monclient: init 2024-03-27T03:08:24.757 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.763+0000 7f0389401640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:24.757 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.763+0000 7f0389401640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:08:24.757 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.763+0000 7f0389401640 10 monclient(hunting): picked mon.noname-b con 0x7f03840a9ac0 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:24.757 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.763+0000 7f0389401640 10 monclient(hunting): picked mon.noname-a con 0x7f03840af900 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:24.757 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.764+0000 7f0389401640 10 monclient(hunting): picked mon.noname-c con 0x7f038413ee90 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:24.757 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.764+0000 7f0389401640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0384071b00 con 0x7f03840af900 2024-03-27T03:08:24.757 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.764+0000 7f0389401640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03840705b0 con 0x7f03840a9ac0 2024-03-27T03:08:24.757 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.764+0000 7f0389401640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0384070a40 con 0x7f038413ee90 2024-03-27T03:08:24.757 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.764+0000 7f0389401640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:24.757 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.764+0000 7f0389401640 10 monclient(hunting): authenticate will time out at 1465.069618s 2024-03-27T03:08:24.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.769+0000 7f0382ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f03840a9ac0 0x7f0384132780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:33993/0 (socket says 172.21.15.153:33993) 2024-03-27T03:08:24.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.769+0000 7f0382ffd640 1 -- 172.21.15.153:0/1368645150 learned_addr learned my addr 172.21.15.153:0/1368645150 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:24.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.769+0000 7f036b7fe640 10 client.?.objecter ms_handle_connect 0x7f03840af900 2024-03-27T03:08:24.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.769+0000 7f036b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:24.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.769+0000 7f036b7fe640 10 client.?.objecter ms_handle_connect 0x7f03840a9ac0 2024-03-27T03:08:24.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.769+0000 7f036b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:24.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.769+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782618056 0 0) 0x7f0384071b00 con 0x7f03840af900 2024-03-27T03:08:24.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.769+0000 7f036b7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:08:24.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.769+0000 7f036b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:24.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.769+0000 7f036b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:24.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.769+0000 7f036b7fe640 10 monclient(hunting): my global_id is 4326 2024-03-27T03:08:24.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.769+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0370004eb0 con 0x7f03840af900 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 client.?.objecter ms_handle_connect 0x7f038413ee90 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817298378 0 0) 0x7f0384070a40 con 0x7f038413ee90 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 monclient(hunting): my global_id is 4349 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0384071b00 con 0x7f038413ee90 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3647538180 0 0) 0x7f0370004eb0 con 0x7f03840af900 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0384070a40 con 0x7f03840af900 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0378001820 con 0x7f03840af900 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:08:24.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 monclient(hunting): dump: 2024-03-27T03:08:24.765 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:24.765 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:24.765 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:24.765 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:24.765 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:24.765 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:24.765 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:24.765 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:24.765 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:24.765 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:24.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:08:24.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740810681 0 0) 0x7f03840705b0 con 0x7f03840a9ac0 2024-03-27T03:08:24.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:08:24.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:24.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:24.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 10 monclient(hunting): my global_id is 4336 2024-03-27T03:08:24.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.770+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0370004eb0 con 0x7f03840a9ac0 2024-03-27T03:08:24.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.771+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2346077722 0 0) 0x7f0370004eb0 con 0x7f03840a9ac0 2024-03-27T03:08:24.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.771+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03840705b0 con 0x7f03840a9ac0 2024-03-27T03:08:24.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.771+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1697737707 0 0) 0x7f0384071b00 con 0x7f038413ee90 2024-03-27T03:08:24.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.771+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0370004eb0 con 0x7f038413ee90 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.771+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f037c0017d0 con 0x7f03840a9ac0 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.771+0000 7f036b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.771+0000 7f036b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.771+0000 7f036b7fe640 10 monclient(hunting): dump: 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:24.767 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.771+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f03740018d0 con 0x7f038413ee90 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.771+0000 7f036b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.771+0000 7f036b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.771+0000 7f036b7fe640 10 monclient(hunting): dump: 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:24.768 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:24.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1594505893 0 0) 0x7f0370004eb0 con 0x7f038413ee90 2024-03-27T03:08:24.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 >> v1:172.21.15.196:6789/0 conn(0x7f03840a9ac0 legacy=0x7f0384132780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 >> v1:172.21.15.153:6789/0 conn(0x7f03840af900 legacy=0x7f03841d7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f036b7fe640 10 monclient: _finish_hunting 0 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f036b7fe640 1 monclient: found mon.c 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f036b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f036b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0384132e90 con 0x7f038413ee90 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f036b7fe640 10 monclient: _finish_auth 0 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f036b7fe640 10 monclient: _check_auth_tickets 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f036b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f0389401640 5 monclient: authenticate success, global_id 4349 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f0389401640 10 monclient: _renew_subs 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f0389401640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f0389401640 1 -- 172.21.15.153:0/1368645150 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03841330e0 con 0x7f038413ee90 2024-03-27T03:08:24.773 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f0389401640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f0389401640 10 monclient: _renew_subs 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f0389401640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f0389401640 1 -- 172.21.15.153:0/1368645150 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0384070120 con 0x7f038413ee90 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.774+0000 7f0389401640 1 librados: init done 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.776+0000 7f0389401640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.776+0000 7f0389401640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.776+0000 7f0389401640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.776+0000 7f0389401640 1 -- 172.21.15.153:0/1368645150 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0384075f10 con 0x7f038413ee90 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.776+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0374001a30 con 0x7f038413ee90 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.776+0000 7f036b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.776+0000 7f03817fa640 4 set_mon_vals no callback set 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.778+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f03740050e0 con 0x7f038413ee90 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.778+0000 7f036b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:24.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.778+0000 7f036b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.778+0000 7f036b7fe640 10 monclient: dump: 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.778+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f037401b0b0 con 0x7f038413ee90 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.779+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f0374084c80 con 0x7f038413ee90 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.779+0000 7f036b7fe640 10 client.4349.objecter ms_dispatch 0x7f0384000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:08:24.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.779+0000 7f036b7fe640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:08:24.776 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.779+0000 7f036b7fe640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:08:24.776 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.779+0000 7f036b7fe640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-27T03:08:24.776 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.779+0000 7f0382ffd640 10 monclient: get_auth_request con 0x7f037006e0c0 auth_method 0 2024-03-27T03:08:24.776 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.780+0000 7f036b7fe640 10 client.4349.objecter ms_handle_connect 0x7f037006e0c0 2024-03-27T03:08:24.778 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.785+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0374084e60 con 0x7f038413ee90 2024-03-27T03:08:24.778 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.785+0000 7f036b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:24.778 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:24.785+0000 7f036b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:08:25.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.106+0000 7f0389401640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:25.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.106+0000 7f0389401640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:25.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.106+0000 7f0389401640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:08:25.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.106+0000 7f0389401640 1 -- 172.21.15.153:0/1368645150 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03840703d0 con 0x7f038413ee90 2024-03-27T03:08:25.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.107+0000 7f036b7fe640 1 -- 172.21.15.153:0/1368645150 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f0374050560 con 0x7f038413ee90 2024-03-27T03:08:25.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.107+0000 7f036b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:25.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.107+0000 7f036b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:08:25.102 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:08:25.103 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:08:25.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.113+0000 7f0389401640 10 librados: watch_flush enter 2024-03-27T03:08:25.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.114+0000 7f0389401640 10 librados: watch_flush exit 2024-03-27T03:08:25.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.114+0000 7f0389401640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-27T03:08:25.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.114+0000 7f0389401640 10 client.4349.objecter successfully canceled tick 2024-03-27T03:08:25.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.114+0000 7f0389401640 1 -- 172.21.15.153:0/1368645150 >> v1:172.21.15.153:6816/2637937781 conn(0x7f037006e0c0 legacy=0x7f0370070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:25.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.114+0000 7f0389401640 10 monclient: shutdown 2024-03-27T03:08:25.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.114+0000 7f0389401640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:25.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.114+0000 7f0389401640 1 -- 172.21.15.153:0/1368645150 >> v1:172.21.15.153:6790/0 conn(0x7f038413ee90 legacy=0x7f03841e2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:25.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.114+0000 7f0389401640 1 -- 172.21.15.153:0/1368645150 shutdown_connections 2024-03-27T03:08:25.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.114+0000 7f0389401640 1 -- 172.21.15.153:0/1368645150 >> 172.21.15.153:0/1368645150 conn(0x7f03840b35e0 msgr2=0x7f03840adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:25.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.114+0000 7f0389401640 1 -- 172.21.15.153:0/1368645150 shutdown_connections 2024-03-27T03:08:25.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.114+0000 7f0389401640 1 -- 172.21.15.153:0/1368645150 wait complete. 2024-03-27T03:08:25.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:25.115+0000 7f0389401640 1 librados: shutdown 2024-03-27T03:08:25.125 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:08:31.932 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:08:32.091 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.098+0000 7fc2d2554640 10 monclient: get_monmap_and_config 2024-03-27T03:08:32.091 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.098+0000 7fc2d2554640 10 monclient: build_initial_monmap 2024-03-27T03:08:32.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.098+0000 7fc2d2554640 10 monclient: monmap: 2024-03-27T03:08:32.092 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:32.092 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:32.092 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:08:32.099277+0000 2024-03-27T03:08:32.092 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:08:32.099277+0000 2024-03-27T03:08:32.092 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:32.093 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:32.093 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:32.093 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:08:32.093 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:08:32.093 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:32.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.099+0000 7fc2d2554640 1 Processor -- start 2024-03-27T03:08:32.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.099+0000 7fc2d2554640 1 -- start start 2024-03-27T03:08:32.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.099+0000 7fc2d2554640 10 monclient: init 2024-03-27T03:08:32.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.100+0000 7fc2d2554640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:32.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.100+0000 7fc2d2554640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:08:32.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.101+0000 7fc2d2554640 10 monclient(hunting): picked mon.noname-c con 0x7fc2cc1457f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:32.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.101+0000 7fc2d2554640 10 monclient(hunting): picked mon.noname-b con 0x7fc2cc13f650 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:32.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.101+0000 7fc2d2554640 10 monclient(hunting): picked mon.noname-a con 0x7fc2cc0a9d70 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:32.097 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.101+0000 7fc2d2554640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc0aaef0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.097 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.101+0000 7fc2d2554640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc145bd0 con 0x7fc2cc1457f0 2024-03-27T03:08:32.097 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.101+0000 7fc2d2554640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc145eb0 con 0x7fc2cc13f650 2024-03-27T03:08:32.097 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.101+0000 7fc2d2554640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:32.097 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.101+0000 7fc2d2554640 10 monclient(hunting): authenticate will time out at 1472.407163s 2024-03-27T03:08:32.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.104+0000 7fc2d0af1640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc2cc0a9d70 0x7fc2cc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56087/0 (socket says 172.21.15.153:56087) 2024-03-27T03:08:32.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.104+0000 7fc2d0af1640 1 -- 172.21.15.153:0/2566668408 learned_addr learned my addr 172.21.15.153:0/2566668408 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:32.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.104+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64342275 0 0) 0x7fc2cc0aaef0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.104+0000 7fc2caffd640 10 monclient(hunting): _negotiate 2024-03-27T03:08:32.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.104+0000 7fc2caffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:32.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.105+0000 7fc2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:32.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.105+0000 7fc2caffd640 10 monclient(hunting): my global_id is 4329 2024-03-27T03:08:32.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.105+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2b4003520 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.103 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245065705 0 0) 0x7fc2cc145eb0 con 0x7fc2cc13f650 2024-03-27T03:08:32.103 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient(hunting): _negotiate 2024-03-27T03:08:32.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:32.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:32.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient(hunting): my global_id is 4352 2024-03-27T03:08:32.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2cc0aaef0 con 0x7fc2cc13f650 2024-03-27T03:08:32.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215226185 0 0) 0x7fc2cc145bd0 con 0x7fc2cc1457f0 2024-03-27T03:08:32.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient(hunting): _negotiate 2024-03-27T03:08:32.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:32.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:32.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient(hunting): my global_id is 4339 2024-03-27T03:08:32.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2cc145eb0 con 0x7fc2cc1457f0 2024-03-27T03:08:32.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3883828898 0 0) 0x7fc2b4003520 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 >> v1:172.21.15.153:6790/0 conn(0x7fc2cc13f650 legacy=0x7fc2cc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:32.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 >> v1:172.21.15.196:6789/0 conn(0x7fc2cc1457f0 legacy=0x7fc2cc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:32.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient: _finish_hunting 0 2024-03-27T03:08:32.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 1 monclient: found mon.noname-a 2024-03-27T03:08:32.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:32.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:08:32.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc146340 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient: _finish_auth 0 2024-03-27T03:08:32.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient: _check_auth_tickets 2024-03-27T03:08:32.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:32.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc2c4001a10 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:32.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:08:32.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 10 monclient: dump: 2024-03-27T03:08:32.106 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:32.107 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:32.107 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:32.107 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:32.107 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:32.107 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:32.107 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:32.107 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:32.107 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:32.108 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:32.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2d2554640 5 monclient: authenticate success, global_id 4329 2024-03-27T03:08:32.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2d2554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:08:32.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.107+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2c4001c30 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.108+0000 7fc2caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:32.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.108+0000 7fc2caffd640 1 -- 172.21.15.153:0/2566668408 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc2c4001fc0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.108+0000 7fc2d2554640 10 monclient: get_monmap_and_config success 2024-03-27T03:08:32.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.108+0000 7fc2d2554640 4 set_mon_vals no callback set 2024-03-27T03:08:32.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.108+0000 7fc2caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:32.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.108+0000 7fc2d2554640 10 monclient: shutdown 2024-03-27T03:08:32.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.108+0000 7fc2caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:08:32.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.108+0000 7fc2caffd640 10 monclient: dump: 2024-03-27T03:08:32.109 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:32.109 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:32.109 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:32.109 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:32.109 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:32.110 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:32.110 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:32.110 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:32.110 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:32.110 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:32.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.108+0000 7fc2d2554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:32.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.108+0000 7fc2d2554640 1 -- 172.21.15.153:0/2566668408 >> v1:172.21.15.153:6789/0 conn(0x7fc2cc0a9d70 legacy=0x7fc2cc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:32.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.109+0000 7fc2d2554640 1 -- 172.21.15.153:0/2566668408 shutdown_connections 2024-03-27T03:08:32.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.109+0000 7fc2d2554640 1 -- 172.21.15.153:0/2566668408 >> 172.21.15.153:0/2566668408 conn(0x7fc2cc066a00 msgr2=0x7fc2cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:32.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.109+0000 7fc2d2554640 1 -- 172.21.15.153:0/2566668408 shutdown_connections 2024-03-27T03:08:32.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.110+0000 7fc2d2554640 1 -- 172.21.15.153:0/2566668408 wait complete. 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.111+0000 7fc2d2554640 10 monclient: build_initial_monmap 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.111+0000 7fc2d2554640 10 monclient: monmap: 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.111+0000 7fc2d2554640 1 librados: starting msgr at 2024-03-27T03:08:32.112 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.111+0000 7fc2d2554640 1 librados: starting objecter 2024-03-27T03:08:32.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.112+0000 7fc2d2554640 1 Processor -- start 2024-03-27T03:08:32.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.112+0000 7fc2d2554640 1 -- start start 2024-03-27T03:08:32.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.112+0000 7fc2d2554640 1 librados: setting wanted keys 2024-03-27T03:08:32.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.112+0000 7fc2d2554640 1 librados: calling monclient init 2024-03-27T03:08:32.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.112+0000 7fc2d2554640 10 monclient: init 2024-03-27T03:08:32.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.113+0000 7fc2d2554640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:32.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.113+0000 7fc2d2554640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:08:32.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.116+0000 7fc2d2554640 10 monclient(hunting): picked mon.noname-b con 0x7fc2cc0a9d70 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:32.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.116+0000 7fc2d2554640 10 monclient(hunting): picked mon.noname-a con 0x7fc2cc13f650 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:32.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.119+0000 7fc2d2554640 10 monclient(hunting): picked mon.noname-c con 0x7fc2cc13b5b0 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.119+0000 7fc2d2554640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc138a90 con 0x7fc2cc13f650 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.119+0000 7fc2d2554640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc13a7a0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.119+0000 7fc2d2554640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc13aa80 con 0x7fc2cc13b5b0 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.119+0000 7fc2d2554640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.119+0000 7fc2d2554640 10 monclient(hunting): authenticate will time out at 1472.425115s 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2cbfff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc2cc0a9d70 0x7fc2cc139200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:50801/0 (socket says 172.21.15.153:50801) 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2cbfff640 1 -- 172.21.15.153:0/3948362969 learned_addr learned my addr 172.21.15.153:0/3948362969 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 10 client.?.objecter ms_handle_connect 0x7fc2cc0a9d70 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518197485 0 0) 0x7fc2cc13a7a0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 10 monclient(hunting): my global_id is 4342 2024-03-27T03:08:32.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2b8004eb0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 10 client.?.objecter ms_handle_connect 0x7fc2cc13f650 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169946992 0 0) 0x7fc2cc138a90 con 0x7fc2cc13f650 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 10 monclient(hunting): my global_id is 4332 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.120+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2cc13a7a0 con 0x7fc2cc13f650 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3814489461 0 0) 0x7fc2cc13a7a0 con 0x7fc2cc13f650 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2cc138a90 con 0x7fc2cc13f650 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc2c00018d0 con 0x7fc2cc13f650 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 10 monclient(hunting): dump: 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:32.117 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1399298749 0 0) 0x7fc2b8004eb0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2cc13a7a0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc2bc0018f0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-27T03:08:32.118 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.121+0000 7fc2c8ff9640 10 monclient(hunting): dump: 2024-03-27T03:08:32.119 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:32.119 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:32.119 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:32.119 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:32.119 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:32.119 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:32.119 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:32.119 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:32.119 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:32.119 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:32.119 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.122+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2986161808 0 0) 0x7fc2cc13a7a0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.119 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.122+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 >> v1:172.21.15.153:6790/0 conn(0x7fc2cc13b5b0 legacy=0x7fc2cc1e1a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:08:32.123 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.123+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 >> v1:172.21.15.153:6789/0 conn(0x7fc2cc13f650 legacy=0x7fc2cc13aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:32.123 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.123+0000 7fc2c8ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:08:32.123 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2c8ff9640 1 monclient: found mon.b 2024-03-27T03:08:32.123 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:32.123 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2c8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:32.123 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc138ce0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.123 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2c8ff9640 10 monclient: _finish_auth 0 2024-03-27T03:08:32.123 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2c8ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2d2554640 5 monclient: authenticate success, global_id 4342 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2d2554640 10 monclient: _renew_subs 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2d2554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2d2554640 1 -- 172.21.15.153:0/3948362969 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2cc1e3130 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2d2554640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2d2554640 10 monclient: _renew_subs 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2d2554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2d2554640 1 -- 172.21.15.153:0/3948362969 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2cc1e34a0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.124+0000 7fc2d2554640 1 librados: init done 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.125+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2bc003610 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.125+0000 7fc2c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.125+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc2bc0039e0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.125+0000 7fc2c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.125+0000 7fc2c8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:08:32.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.125+0000 7fc2c8ff9640 10 monclient: dump: 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.126+0000 7fc2ca7fc640 4 set_mon_vals no callback set 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.127+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fc2bc003b40 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.128+0000 7fc2d2554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.128+0000 7fc2d2554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:32.125 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.128+0000 7fc2d2554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:32.126 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.128+0000 7fc2d2554640 1 -- 172.21.15.153:0/3948362969 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ac005080 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.126 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.129+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fc2bc0849c0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.126 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.129+0000 7fc2c8ff9640 10 client.4342.objecter ms_dispatch 0x7fc2cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:08:32.126 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.129+0000 7fc2c8ff9640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:08:32.126 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.129+0000 7fc2c8ff9640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:08:32.126 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.129+0000 7fc2cb7fe640 10 monclient: get_auth_request con 0x7fc2b806de30 auth_method 0 2024-03-27T03:08:32.126 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.129+0000 7fc2c8ff9640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-27T03:08:32.126 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.129+0000 7fc2c8ff9640 10 client.4342.objecter ms_handle_connect 0x7fc2b806de30 2024-03-27T03:08:32.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.138+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2bc0502a0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.138+0000 7fc2c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:32.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.138+0000 7fc2c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:08:32.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.459+0000 7fc2d2554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:32.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.460+0000 7fc2d2554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:32.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.460+0000 7fc2d2554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:32.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.460+0000 7fc2d2554640 1 -- 172.21.15.153:0/3948362969 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2ac005330 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.464+0000 7fc2c8ff9640 1 -- 172.21.15.153:0/3948362969 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fc2bc0521d0 con 0x7fc2cc0a9d70 2024-03-27T03:08:32.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.464+0000 7fc2c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:32.458 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.464+0000 7fc2c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:08:32.460 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:08:32.460 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:08:32.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.471+0000 7fc2d2554640 10 librados: watch_flush enter 2024-03-27T03:08:32.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.471+0000 7fc2d2554640 10 librados: watch_flush exit 2024-03-27T03:08:32.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.471+0000 7fc2d2554640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-27T03:08:32.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.471+0000 7fc2d2554640 10 client.4342.objecter successfully canceled tick 2024-03-27T03:08:32.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.471+0000 7fc2d2554640 1 -- 172.21.15.153:0/3948362969 >> v1:172.21.15.153:6816/2637937781 conn(0x7fc2b806de30 legacy=0x7fc2b80702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:32.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.471+0000 7fc2d2554640 10 monclient: shutdown 2024-03-27T03:08:32.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.471+0000 7fc2d2554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:32.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.471+0000 7fc2d2554640 1 -- 172.21.15.153:0/3948362969 >> v1:172.21.15.196:6789/0 conn(0x7fc2cc0a9d70 legacy=0x7fc2cc139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:32.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.471+0000 7fc2d2554640 1 -- 172.21.15.153:0/3948362969 shutdown_connections 2024-03-27T03:08:32.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.471+0000 7fc2d2554640 1 -- 172.21.15.153:0/3948362969 >> 172.21.15.153:0/3948362969 conn(0x7fc2cc066a00 msgr2=0x7fc2cc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:32.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.472+0000 7fc2d2554640 1 -- 172.21.15.153:0/3948362969 shutdown_connections 2024-03-27T03:08:32.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.472+0000 7fc2d2554640 1 -- 172.21.15.153:0/3948362969 wait complete. 2024-03-27T03:08:32.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:32.472+0000 7fc2d2554640 1 librados: shutdown 2024-03-27T03:08:32.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:08:39.289 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:08:39.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.453+0000 7f3c8a76d640 10 monclient: get_monmap_and_config 2024-03-27T03:08:39.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.453+0000 7f3c8a76d640 10 monclient: build_initial_monmap 2024-03-27T03:08:39.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.454+0000 7f3c8a76d640 10 monclient: monmap: 2024-03-27T03:08:39.448 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:39.448 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:39.448 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:08:39.454901+0000 2024-03-27T03:08:39.448 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:08:39.454901+0000 2024-03-27T03:08:39.448 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:39.448 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:39.448 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:39.448 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:08:39.448 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:08:39.448 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:39.449 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.456+0000 7f3c8a76d640 1 Processor -- start 2024-03-27T03:08:39.449 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.456+0000 7f3c8a76d640 1 -- start start 2024-03-27T03:08:39.449 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.456+0000 7f3c8a76d640 10 monclient: init 2024-03-27T03:08:39.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.457+0000 7f3c8a76d640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:39.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.457+0000 7f3c8a76d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:08:39.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.457+0000 7f3c8a76d640 10 monclient(hunting): picked mon.noname-c con 0x7f3c840ae250 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:39.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.457+0000 7f3c8a76d640 10 monclient(hunting): picked mon.noname-b con 0x7f3c8413f970 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:39.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.457+0000 7f3c8a76d640 10 monclient(hunting): picked mon.noname-a con 0x7f3c84142800 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:39.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.457+0000 7f3c8a76d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c840afe00 con 0x7f3c84142800 2024-03-27T03:08:39.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.457+0000 7f3c8a76d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c84142c10 con 0x7f3c840ae250 2024-03-27T03:08:39.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.457+0000 7f3c8a76d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c840afb20 con 0x7f3c8413f970 2024-03-27T03:08:39.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.457+0000 7f3c8a76d640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:39.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.457+0000 7f3c8a76d640 10 monclient(hunting): authenticate will time out at 1479.762934s 2024-03-27T03:08:39.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.460+0000 7f3c88d0a640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3c84142800 0x7f3c8414da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:33201/0 (socket says 172.21.15.153:33201) 2024-03-27T03:08:39.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.460+0000 7f3c88d0a640 1 -- 172.21.15.153:0/2499712054 learned_addr learned my addr 172.21.15.153:0/2499712054 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:39.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.461+0000 7f3c82ffd640 1 -- 172.21.15.153:0/2499712054 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239606651 0 0) 0x7f3c840afe00 con 0x7f3c84142800 2024-03-27T03:08:39.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.461+0000 7f3c82ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:08:39.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.461+0000 7f3c82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:39.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.461+0000 7f3c82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:39.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.461+0000 7f3c82ffd640 10 monclient(hunting): my global_id is 4335 2024-03-27T03:08:39.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.461+0000 7f3c82ffd640 1 -- 172.21.15.153:0/2499712054 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c70003520 con 0x7f3c84142800 2024-03-27T03:08:39.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.461+0000 7f3c82ffd640 1 -- 172.21.15.153:0/2499712054 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1568153446 0 0) 0x7f3c70003520 con 0x7f3c84142800 2024-03-27T03:08:39.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.461+0000 7f3c82ffd640 1 -- 172.21.15.153:0/2499712054 >> v1:172.21.15.153:6790/0 conn(0x7f3c8413f970 legacy=0x7f3c841420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:08:39.461 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 1 -- 172.21.15.153:0/2499712054 >> v1:172.21.15.196:6789/0 conn(0x7f3c840ae250 legacy=0x7f3c8413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:39.461 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 1 monclient: found mon.noname-a 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 1 -- 172.21.15.153:0/2499712054 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c840aab80 con 0x7f3c84142800 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 10 monclient: _finish_auth 0 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 1 -- 172.21.15.153:0/2499712054 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3c6c001690 con 0x7f3c84142800 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c82ffd640 10 monclient: dump: 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:39.462 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c8a76d640 5 monclient: authenticate success, global_id 4335 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.466+0000 7f3c8a76d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.467+0000 7f3c82ffd640 1 -- 172.21.15.153:0/2499712054 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c6c0018f0 con 0x7f3c84142800 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.467+0000 7f3c82ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.467+0000 7f3c8a76d640 10 monclient: get_monmap_and_config success 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.467+0000 7f3c8a76d640 4 set_mon_vals no callback set 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.467+0000 7f3c8a76d640 10 monclient: shutdown 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.468+0000 7f3c8a76d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:39.463 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.468+0000 7f3c8a76d640 1 -- 172.21.15.153:0/2499712054 >> v1:172.21.15.153:6789/0 conn(0x7f3c84142800 legacy=0x7f3c8414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:39.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.468+0000 7f3c82ffd640 1 -- 172.21.15.153:0/2499712054 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3c6c001d00 con 0x7f3c84142800 2024-03-27T03:08:39.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.468+0000 7f3c82ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-27T03:08:39.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.470+0000 7f3c8a76d640 1 -- 172.21.15.153:0/2499712054 shutdown_connections 2024-03-27T03:08:39.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.470+0000 7f3c8a76d640 1 -- 172.21.15.153:0/2499712054 >> 172.21.15.153:0/2499712054 conn(0x7f3c840b35e0 msgr2=0x7f3c840b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:39.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.471+0000 7f3c8a76d640 1 -- 172.21.15.153:0/2499712054 shutdown_connections 2024-03-27T03:08:39.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.471+0000 7f3c8a76d640 1 -- 172.21.15.153:0/2499712054 wait complete. 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.472+0000 7f3c8a76d640 10 monclient: build_initial_monmap 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.472+0000 7f3c8a76d640 10 monclient: monmap: 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:39.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.473+0000 7f3c8a76d640 1 librados: starting msgr at 2024-03-27T03:08:39.467 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.473+0000 7f3c8a76d640 1 librados: starting objecter 2024-03-27T03:08:39.467 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.473+0000 7f3c8a76d640 1 Processor -- start 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.473+0000 7f3c8a76d640 1 -- start start 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.473+0000 7f3c8a76d640 1 librados: setting wanted keys 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.473+0000 7f3c8a76d640 1 librados: calling monclient init 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.473+0000 7f3c8a76d640 10 monclient: init 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.474+0000 7f3c8a76d640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.474+0000 7f3c8a76d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.474+0000 7f3c8a76d640 10 monclient(hunting): picked mon.noname-c con 0x7f3c840ae250 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.474+0000 7f3c8a76d640 10 monclient(hunting): picked mon.noname-b con 0x7f3c8413f970 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.474+0000 7f3c8a76d640 10 monclient(hunting): picked mon.noname-a con 0x7f3c84142800 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.474+0000 7f3c8a76d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c84143b70 con 0x7f3c84142800 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.474+0000 7f3c8a76d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c840ac430 con 0x7f3c8413f970 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.474+0000 7f3c8a76d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c840ad610 con 0x7f3c840ae250 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.474+0000 7f3c8a76d640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:39.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.474+0000 7f3c8a76d640 10 monclient(hunting): authenticate will time out at 1479.780050s 2024-03-27T03:08:39.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.478+0000 7f3c837fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3c8413f970 0x7f3c84140f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:43789/0 (socket says 172.21.15.153:43789) 2024-03-27T03:08:39.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.478+0000 7f3c837fe640 1 -- 172.21.15.153:0/1386887360 learned_addr learned my addr 172.21.15.153:0/1386887360 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:39.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.479+0000 7f3c80ff9640 10 client.?.objecter ms_handle_connect 0x7f3c8413f970 2024-03-27T03:08:39.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.479+0000 7f3c80ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:39.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.479+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136035730 0 0) 0x7f3c840ac430 con 0x7f3c8413f970 2024-03-27T03:08:39.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.479+0000 7f3c80ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:08:39.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.479+0000 7f3c80ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:39.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.479+0000 7f3c80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:39.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.479+0000 7f3c80ff9640 10 monclient(hunting): my global_id is 4348 2024-03-27T03:08:39.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.479+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c64004eb0 con 0x7f3c8413f970 2024-03-27T03:08:39.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.483+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1244795352 0 0) 0x7f3c64004eb0 con 0x7f3c8413f970 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.483+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c840ac430 con 0x7f3c8413f970 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.483+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3c74004080 con 0x7f3c8413f970 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.483+0000 7f3c80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.483+0000 7f3c80ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.483+0000 7f3c80ff9640 10 monclient(hunting): dump: 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:39.477 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:39.478 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.483+0000 7f3c80ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:08:39.478 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.483+0000 7f3c80ff9640 10 client.?.objecter ms_handle_connect 0x7f3c84142800 2024-03-27T03:08:39.478 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.483+0000 7f3c80ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:39.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.485+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138037913 0 0) 0x7f3c84143b70 con 0x7f3c84142800 2024-03-27T03:08:39.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.485+0000 7f3c80ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:08:39.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.485+0000 7f3c80ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:39.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.485+0000 7f3c80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:39.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.485+0000 7f3c80ff9640 10 monclient(hunting): my global_id is 4338 2024-03-27T03:08:39.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.485+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c64004eb0 con 0x7f3c84142800 2024-03-27T03:08:39.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.485+0000 7f3c80ff9640 10 client.?.objecter ms_handle_connect 0x7f3c840ae250 2024-03-27T03:08:39.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.485+0000 7f3c80ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:39.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4060918371 0 0) 0x7f3c64004eb0 con 0x7f3c84142800 2024-03-27T03:08:39.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c84143b70 con 0x7f3c84142800 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3c6c0014b0 con 0x7f3c84142800 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 10 monclient(hunting): dump: 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:39.480 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121624018 0 0) 0x7f3c840ad610 con 0x7f3c840ae250 2024-03-27T03:08:39.481 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:08:39.481 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:39.481 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:39.481 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 10 monclient(hunting): my global_id is 4355 2024-03-27T03:08:39.481 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c64004eb0 con 0x7f3c840ae250 2024-03-27T03:08:39.481 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3469653944 0 0) 0x7f3c84143b70 con 0x7f3c84142800 2024-03-27T03:08:39.481 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 >> v1:172.21.15.153:6790/0 conn(0x7f3c840ae250 legacy=0x7f3c841dec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:39.481 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 >> v1:172.21.15.196:6789/0 conn(0x7f3c8413f970 legacy=0x7f3c84140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:39.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:08:39.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 1 monclient: found mon.a 2024-03-27T03:08:39.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:39.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c84143470 con 0x7f3c84142800 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 10 monclient: _finish_auth 0 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c8a76d640 5 monclient: authenticate success, global_id 4338 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c8a76d640 10 monclient: _renew_subs 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c8a76d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c8a76d640 1 -- 172.21.15.153:0/1386887360 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c84143750 con 0x7f3c84142800 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c8a76d640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c8a76d640 10 monclient: _renew_subs 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c8a76d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c8a76d640 1 -- 172.21.15.153:0/1386887360 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c841df350 con 0x7f3c84142800 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.486+0000 7f3c8a76d640 1 librados: init done 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.488+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c6c004a90 con 0x7f3c84142800 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.488+0000 7f3c80ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:39.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.488+0000 7f3c827fc640 4 set_mon_vals no callback set 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.489+0000 7f3c8a76d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.489+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3c6c004da0 con 0x7f3c84142800 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.489+0000 7f3c8a76d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.489+0000 7f3c8a76d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.489+0000 7f3c8a76d640 1 -- 172.21.15.153:0/1386887360 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c68005080 con 0x7f3c84142800 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.490+0000 7f3c80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.490+0000 7f3c80ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.490+0000 7f3c80ff9640 10 monclient: dump: 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:39.485 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:39.486 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:39.486 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:39.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.493+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f3c6c01ad70 con 0x7f3c84142800 2024-03-27T03:08:39.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.493+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f3c6c084920 con 0x7f3c84142800 2024-03-27T03:08:39.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.493+0000 7f3c80ff9640 10 client.4338.objecter ms_dispatch 0x7f3c84000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:08:39.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.494+0000 7f3c80ff9640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:08:39.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.494+0000 7f3c80ff9640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:08:39.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.494+0000 7f3c80ff9640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-27T03:08:39.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.494+0000 7f3c83fff640 10 monclient: get_auth_request con 0x7f3c6406e140 auth_method 0 2024-03-27T03:08:39.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.494+0000 7f3c80ff9640 10 client.4338.objecter ms_handle_connect 0x7f3c6406e140 2024-03-27T03:08:39.493 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.500+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c6c050200 con 0x7f3c84142800 2024-03-27T03:08:39.493 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.500+0000 7f3c80ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:39.493 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.500+0000 7f3c80ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:08:39.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.831+0000 7f3c8a76d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:39.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.831+0000 7f3c8a76d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:39.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.831+0000 7f3c8a76d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:08:39.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.832+0000 7f3c8a76d640 1 -- 172.21.15.153:0/1386887360 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c68005330 con 0x7f3c84142800 2024-03-27T03:08:39.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.834+0000 7f3c80ff9640 1 -- 172.21.15.153:0/1386887360 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f3c6c052130 con 0x7f3c84142800 2024-03-27T03:08:39.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.834+0000 7f3c80ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:39.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.834+0000 7f3c80ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:08:39.828 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:08:39.829 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:08:39.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.838+0000 7f3c8a76d640 10 librados: watch_flush enter 2024-03-27T03:08:39.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.838+0000 7f3c8a76d640 10 librados: watch_flush exit 2024-03-27T03:08:39.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.838+0000 7f3c8a76d640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-27T03:08:39.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.838+0000 7f3c8a76d640 10 client.4338.objecter successfully canceled tick 2024-03-27T03:08:39.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.838+0000 7f3c8a76d640 1 -- 172.21.15.153:0/1386887360 >> v1:172.21.15.153:6816/2637937781 conn(0x7f3c6406e140 legacy=0x7f3c64070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:39.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.838+0000 7f3c8a76d640 10 monclient: shutdown 2024-03-27T03:08:39.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.838+0000 7f3c8a76d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:39.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.838+0000 7f3c8a76d640 1 -- 172.21.15.153:0/1386887360 >> v1:172.21.15.153:6789/0 conn(0x7f3c84142800 legacy=0x7f3c84143fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:39.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.838+0000 7f3c8a76d640 1 -- 172.21.15.153:0/1386887360 shutdown_connections 2024-03-27T03:08:39.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.838+0000 7f3c8a76d640 1 -- 172.21.15.153:0/1386887360 >> 172.21.15.153:0/1386887360 conn(0x7f3c840b35e0 msgr2=0x7f3c840b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:39.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.842+0000 7f3c8a76d640 1 -- 172.21.15.153:0/1386887360 shutdown_connections 2024-03-27T03:08:39.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.842+0000 7f3c8a76d640 1 -- 172.21.15.153:0/1386887360 wait complete. 2024-03-27T03:08:39.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:39.842+0000 7f3c8a76d640 1 librados: shutdown 2024-03-27T03:08:39.853 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:08:46.660 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:08:46.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.821+0000 7f611359e640 10 monclient: get_monmap_and_config 2024-03-27T03:08:46.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.821+0000 7f611359e640 10 monclient: build_initial_monmap 2024-03-27T03:08:46.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.821+0000 7f611359e640 10 monclient: monmap: 2024-03-27T03:08:46.815 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:46.815 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:46.816 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:08:46.822691+0000 2024-03-27T03:08:46.816 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:08:46.822691+0000 2024-03-27T03:08:46.816 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:46.816 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:46.816 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:46.816 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:08:46.816 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:08:46.816 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:46.816 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.823+0000 7f611359e640 1 Processor -- start 2024-03-27T03:08:46.817 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.823+0000 7f611359e640 1 -- start start 2024-03-27T03:08:46.817 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.823+0000 7f611359e640 10 monclient: init 2024-03-27T03:08:46.817 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.824+0000 7f611359e640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:46.817 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.824+0000 7f611359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.824+0000 7f611359e640 10 monclient(hunting): picked mon.noname-b con 0x7f6114071f40 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.824+0000 7f611359e640 10 monclient(hunting): picked mon.noname-a con 0x7f61140adcb0 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.825+0000 7f611359e640 10 monclient(hunting): picked mon.noname-c con 0x7f611413f9a0 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.825+0000 7f611359e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61140af1b0 con 0x7f61140adcb0 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.825+0000 7f611359e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61140b0030 con 0x7f611413f9a0 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.825+0000 7f611359e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61140aec40 con 0x7f6114071f40 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.825+0000 7f611359e640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.825+0000 7f611359e640 10 monclient(hunting): authenticate will time out at 1487.130633s 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.828+0000 7f611259c640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f6114071f40 0x7f611413f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:40513/0 (socket says 172.21.15.153:40513) 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.828+0000 7f611259c640 1 -- 172.21.15.153:0/3485188786 learned_addr learned my addr 172.21.15.153:0/3485188786 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.828+0000 7f611159a640 1 -- 172.21.15.153:0/3485188786 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769391680 0 0) 0x7f61140aec40 con 0x7f6114071f40 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.828+0000 7f611159a640 10 monclient(hunting): _negotiate 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.828+0000 7f611159a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:46.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.828+0000 7f611159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:46.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.828+0000 7f611159a640 10 monclient(hunting): my global_id is 4358 2024-03-27T03:08:46.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.828+0000 7f611159a640 1 -- 172.21.15.153:0/3485188786 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60fc003520 con 0x7f6114071f40 2024-03-27T03:08:46.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 1 -- 172.21.15.153:0/3485188786 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2465613925 0 0) 0x7f60fc003520 con 0x7f6114071f40 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 1 -- 172.21.15.153:0/3485188786 >> v1:172.21.15.196:6789/0 conn(0x7f611413f9a0 legacy=0x7f6114145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 1 -- 172.21.15.153:0/3485188786 >> v1:172.21.15.153:6789/0 conn(0x7f61140adcb0 legacy=0x7f61141457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 10 monclient: _finish_hunting 0 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 1 monclient: found mon.noname-b 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 1 -- 172.21.15.153:0/3485188786 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61140aedd0 con 0x7f6114071f40 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 10 monclient: _finish_auth 0 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 10 monclient: _check_auth_tickets 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 1 -- 172.21.15.153:0/3485188786 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6104001690 con 0x7f6114071f40 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:08:46.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.831+0000 7f611159a640 10 monclient: dump: 2024-03-27T03:08:46.827 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:46.827 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:46.827 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:46.827 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:46.827 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:46.827 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:46.827 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:46.827 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:46.827 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:46.827 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:46.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.832+0000 7f611359e640 5 monclient: authenticate success, global_id 4358 2024-03-27T03:08:46.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.832+0000 7f611359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.833+0000 7f611159a640 1 -- 172.21.15.153:0/3485188786 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6104001860 con 0x7f6114071f40 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.833+0000 7f611159a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.833+0000 7f611159a640 1 -- 172.21.15.153:0/3485188786 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6104001c70 con 0x7f6114071f40 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.833+0000 7f611159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.833+0000 7f611159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.833+0000 7f611159a640 10 monclient: dump: 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:46.829 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:46.830 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:46.830 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:46.830 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:46.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.833+0000 7f611359e640 10 monclient: get_monmap_and_config success 2024-03-27T03:08:46.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.833+0000 7f611359e640 4 set_mon_vals no callback set 2024-03-27T03:08:46.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.834+0000 7f611359e640 10 monclient: shutdown 2024-03-27T03:08:46.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.834+0000 7f611359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:46.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.834+0000 7f611359e640 1 -- 172.21.15.153:0/3485188786 >> v1:172.21.15.153:6790/0 conn(0x7f6114071f40 legacy=0x7f611413f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:46.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.835+0000 7f611359e640 1 -- 172.21.15.153:0/3485188786 shutdown_connections 2024-03-27T03:08:46.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.835+0000 7f611359e640 1 -- 172.21.15.153:0/3485188786 >> 172.21.15.153:0/3485188786 conn(0x7f61140673e0 msgr2=0x7f61140b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:46.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.835+0000 7f611359e640 1 -- 172.21.15.153:0/3485188786 shutdown_connections 2024-03-27T03:08:46.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.835+0000 7f611359e640 1 -- 172.21.15.153:0/3485188786 wait complete. 2024-03-27T03:08:46.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.836+0000 7f611359e640 10 monclient: build_initial_monmap 2024-03-27T03:08:46.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.836+0000 7f611359e640 10 monclient: monmap: 2024-03-27T03:08:46.831 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:46.831 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:08:46.831 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:08:46.831 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:08:46.831 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:46.831 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:46.831 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:46.831 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:08:46.831 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:08:46.832 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:46.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.836+0000 7f611359e640 1 librados: starting msgr at 2024-03-27T03:08:46.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.836+0000 7f611359e640 1 librados: starting objecter 2024-03-27T03:08:46.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.836+0000 7f611359e640 1 Processor -- start 2024-03-27T03:08:46.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.838+0000 7f611359e640 1 -- start start 2024-03-27T03:08:46.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.838+0000 7f611359e640 1 librados: setting wanted keys 2024-03-27T03:08:46.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.838+0000 7f611359e640 1 librados: calling monclient init 2024-03-27T03:08:46.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.838+0000 7f611359e640 10 monclient: init 2024-03-27T03:08:46.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.839+0000 7f611359e640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:46.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.839+0000 7f611359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:08:46.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.839+0000 7f611359e640 10 monclient(hunting): picked mon.noname-a con 0x7f6114071f40 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:46.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.839+0000 7f611359e640 10 monclient(hunting): picked mon.noname-c con 0x7f61140adcb0 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:46.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.839+0000 7f611359e640 10 monclient(hunting): picked mon.noname-b con 0x7f611413f9a0 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:46.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.839+0000 7f611359e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6114140430 con 0x7f6114071f40 2024-03-27T03:08:46.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.839+0000 7f611359e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6114132890 con 0x7f611413f9a0 2024-03-27T03:08:46.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.839+0000 7f611359e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f611413fe00 con 0x7f61140adcb0 2024-03-27T03:08:46.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.839+0000 7f611359e640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:46.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.839+0000 7f611359e640 10 monclient(hunting): authenticate will time out at 1487.145496s 2024-03-27T03:08:46.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.841+0000 7f611259c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6114071f40 0x7f6114144ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:53992/0 (socket says 172.21.15.153:53992) 2024-03-27T03:08:46.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.841+0000 7f611259c640 1 -- 172.21.15.153:0/828033769 learned_addr learned my addr 172.21.15.153:0/828033769 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:46.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.843+0000 7f6102ffd640 10 client.?.objecter ms_handle_connect 0x7f6114071f40 2024-03-27T03:08:46.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.843+0000 7f6102ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:46.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.843+0000 7f6102ffd640 10 client.?.objecter ms_handle_connect 0x7f61140adcb0 2024-03-27T03:08:46.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.843+0000 7f6102ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:46.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.843+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358678808 0 0) 0x7f6114140430 con 0x7f6114071f40 2024-03-27T03:08:46.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.843+0000 7f6102ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:08:46.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.843+0000 7f6102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:46.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.843+0000 7f6102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:46.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.843+0000 7f6102ffd640 10 monclient(hunting): my global_id is 4341 2024-03-27T03:08:46.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.843+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f4004eb0 con 0x7f6114071f40 2024-03-27T03:08:46.846 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 610600791 0 0) 0x7f60f4004eb0 con 0x7f6114071f40 2024-03-27T03:08:46.846 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6114140430 con 0x7f6114071f40 2024-03-27T03:08:46.846 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090856245 0 0) 0x7f611413fe00 con 0x7f61140adcb0 2024-03-27T03:08:46.846 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:08:46.846 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:46.846 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:46.846 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 10 monclient(hunting): my global_id is 4361 2024-03-27T03:08:46.846 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f4004eb0 con 0x7f61140adcb0 2024-03-27T03:08:46.846 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6104001b70 con 0x7f6114071f40 2024-03-27T03:08:46.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:46.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:08:46.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 10 monclient(hunting): dump: 2024-03-27T03:08:46.847 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:46.847 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:46.847 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:46.847 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:46.847 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:46.847 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:46.848 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:46.848 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:46.848 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:46.848 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:46.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:08:46.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 85136772 0 0) 0x7f6114140430 con 0x7f6114071f40 2024-03-27T03:08:46.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 >> v1:172.21.15.153:6790/0 conn(0x7f61140adcb0 legacy=0x7f6114140850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:46.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 >> v1:172.21.15.196:6789/0 conn(0x7f611413f9a0 legacy=0x7f61141e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:46.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:08:46.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 1 monclient: found mon.a 2024-03-27T03:08:46.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:46.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:08:46.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6114131540 con 0x7f6114071f40 2024-03-27T03:08:46.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 10 monclient: _finish_auth 0 2024-03-27T03:08:46.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:08:46.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f6102ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:46.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f611359e640 5 monclient: authenticate success, global_id 4341 2024-03-27T03:08:46.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f611359e640 10 monclient: _renew_subs 2024-03-27T03:08:46.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.846+0000 7f611359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:08:46.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f611359e640 1 -- 172.21.15.153:0/828033769 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6114131790 con 0x7f6114071f40 2024-03-27T03:08:46.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f611359e640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:08:46.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f611359e640 10 monclient: _renew_subs 2024-03-27T03:08:46.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f611359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:08:46.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f611359e640 1 -- 172.21.15.153:0/828033769 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6114132330 con 0x7f6114071f40 2024-03-27T03:08:46.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f611359e640 1 librados: init done 2024-03-27T03:08:46.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6104001cd0 con 0x7f6114071f40 2024-03-27T03:08:46.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f6102ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:46.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6104004d90 con 0x7f6114071f40 2024-03-27T03:08:46.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f6102ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:46.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f6102ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:08:46.851 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f6102ffd640 10 monclient: dump: 2024-03-27T03:08:46.851 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:46.851 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:46.851 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:46.851 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:46.851 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:46.851 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:46.851 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:46.851 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.847+0000 7f6110d99640 4 set_mon_vals no callback set 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.848+0000 7f611359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.848+0000 7f611359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.848+0000 7f611359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.848+0000 7f611359e640 1 -- 172.21.15.153:0/828033769 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f8005080 con 0x7f6114071f40 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.849+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f610401ad60 con 0x7f6114071f40 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.850+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f6104084bf0 con 0x7f6114071f40 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.850+0000 7f6102ffd640 10 client.4341.objecter ms_dispatch 0x7f6114000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.850+0000 7f6102ffd640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.850+0000 7f6102ffd640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:08:46.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.850+0000 7f6102ffd640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-27T03:08:46.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.851+0000 7f6111d9b640 10 monclient: get_auth_request con 0x7f60f406dd70 auth_method 0 2024-03-27T03:08:46.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.851+0000 7f6102ffd640 10 client.4341.objecter ms_handle_connect 0x7f60f406dd70 2024-03-27T03:08:46.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.857+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6104050300 con 0x7f6114071f40 2024-03-27T03:08:46.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.858+0000 7f6102ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:46.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:46.858+0000 7f6102ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:08:47.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.191+0000 7f611359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:47.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.191+0000 7f611359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:47.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.191+0000 7f611359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:08:47.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.191+0000 7f611359e640 1 -- 172.21.15.153:0/828033769 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60f8005330 con 0x7f6114071f40 2024-03-27T03:08:47.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.192+0000 7f6102ffd640 1 -- 172.21.15.153:0/828033769 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f6104052400 con 0x7f6114071f40 2024-03-27T03:08:47.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.192+0000 7f6102ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:47.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.192+0000 7f6102ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:08:47.187 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:08:47.187 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:08:47.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.198+0000 7f611359e640 10 librados: watch_flush enter 2024-03-27T03:08:47.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.198+0000 7f611359e640 10 librados: watch_flush exit 2024-03-27T03:08:47.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.198+0000 7f611359e640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-27T03:08:47.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.198+0000 7f611359e640 10 client.4341.objecter successfully canceled tick 2024-03-27T03:08:47.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.198+0000 7f611359e640 1 -- 172.21.15.153:0/828033769 >> v1:172.21.15.153:6816/2637937781 conn(0x7f60f406dd70 legacy=0x7f60f4070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:47.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.198+0000 7f611359e640 10 monclient: shutdown 2024-03-27T03:08:47.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.198+0000 7f611359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:47.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.198+0000 7f611359e640 1 -- 172.21.15.153:0/828033769 >> v1:172.21.15.153:6789/0 conn(0x7f6114071f40 legacy=0x7f6114144ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:47.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.199+0000 7f611359e640 1 -- 172.21.15.153:0/828033769 shutdown_connections 2024-03-27T03:08:47.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.199+0000 7f611359e640 1 -- 172.21.15.153:0/828033769 >> 172.21.15.153:0/828033769 conn(0x7f61140673e0 msgr2=0x7f6114069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:47.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.199+0000 7f611359e640 1 -- 172.21.15.153:0/828033769 shutdown_connections 2024-03-27T03:08:47.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.199+0000 7f611359e640 1 -- 172.21.15.153:0/828033769 wait complete. 2024-03-27T03:08:47.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:47.199+0000 7f611359e640 1 librados: shutdown 2024-03-27T03:08:47.210 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:08:54.017 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:08:54.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.173+0000 7f0d85579640 10 monclient: get_monmap_and_config 2024-03-27T03:08:54.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.173+0000 7f0d85579640 10 monclient: build_initial_monmap 2024-03-27T03:08:54.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.173+0000 7f0d85579640 10 monclient: monmap: 2024-03-27T03:08:54.167 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:54.167 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:54.167 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:08:54.175123+0000 2024-03-27T03:08:54.167 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:08:54.175123+0000 2024-03-27T03:08:54.167 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:54.167 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:54.168 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:54.168 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:08:54.168 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:08:54.168 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:54.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.174+0000 7f0d85579640 1 Processor -- start 2024-03-27T03:08:54.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.174+0000 7f0d85579640 1 -- start start 2024-03-27T03:08:54.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.174+0000 7f0d85579640 10 monclient: init 2024-03-27T03:08:54.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.175+0000 7f0d85579640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:54.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.175+0000 7f0d85579640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:08:54.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.175+0000 7f0d85579640 10 monclient(hunting): picked mon.noname-a con 0x7f0d801422b0 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:54.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.175+0000 7f0d85579640 10 monclient(hunting): picked mon.noname-c con 0x7f0d80142690 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:54.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.175+0000 7f0d85579640 10 monclient(hunting): picked mon.noname-b con 0x7f0d800b17d0 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:54.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.175+0000 7f0d7effd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0d801422b0 0x7f0d8013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44950/0 (socket says 172.21.15.153:44950) 2024-03-27T03:08:54.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d7effd640 1 -- 172.21.15.153:0/3559200076 learned_addr learned my addr 172.21.15.153:0/3559200076 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:54.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d85579640 1 -- 172.21.15.153:0/3559200076 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d80142c80 con 0x7f0d801422b0 2024-03-27T03:08:54.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d85579640 1 -- 172.21.15.153:0/3559200076 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d800aae40 con 0x7f0d80142690 2024-03-27T03:08:54.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d85579640 1 -- 172.21.15.153:0/3559200076 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8013fbd0 con 0x7f0d800b17d0 2024-03-27T03:08:54.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d85579640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:54.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d85579640 10 monclient(hunting): authenticate will time out at 1494.482814s 2024-03-27T03:08:54.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202066592 0 0) 0x7f0d800aae40 con 0x7f0d80142690 2024-03-27T03:08:54.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d7dffb640 10 monclient(hunting): _negotiate 2024-03-27T03:08:54.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:54.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d7dffb640 10 monclient(hunting): my global_id is 4351 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d6c003520 con 0x7f0d80142690 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.176+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575881552 0 0) 0x7f0d8013fbd0 con 0x7f0d800b17d0 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient(hunting): _negotiate 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient(hunting): my global_id is 4364 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d800aae40 con 0x7f0d800b17d0 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350921795 0 0) 0x7f0d80142c80 con 0x7f0d801422b0 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient(hunting): _negotiate 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:54.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient(hunting): my global_id is 4344 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d8013fbd0 con 0x7f0d801422b0 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4178985708 0 0) 0x7f0d6c003520 con 0x7f0d80142690 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 >> v1:172.21.15.153:6790/0 conn(0x7f0d800b17d0 legacy=0x7f0d800b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 >> v1:172.21.15.153:6789/0 conn(0x7f0d801422b0 legacy=0x7f0d8013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient: _finish_hunting 0 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 1 monclient: found mon.noname-c 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d80151940 con 0x7f0d80142690 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient: _finish_auth 0 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient: _check_auth_tickets 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:54.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0d78001990 con 0x7f0d80142690 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.177+0000 7f0d7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.178+0000 7f0d7dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.178+0000 7f0d7dffb640 10 monclient: dump: 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.178+0000 7f0d85579640 5 monclient: authenticate success, global_id 4351 2024-03-27T03:08:54.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.178+0000 7f0d85579640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:08:54.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.178+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d78001bb0 con 0x7f0d80142690 2024-03-27T03:08:54.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.178+0000 7f0d7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:54.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.178+0000 7f0d7dffb640 1 -- 172.21.15.153:0/3559200076 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0d78001f40 con 0x7f0d80142690 2024-03-27T03:08:54.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.178+0000 7f0d7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:54.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.178+0000 7f0d7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:08:54.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.178+0000 7f0d7dffb640 10 monclient: dump: 2024-03-27T03:08:54.178 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:54.178 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:54.178 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:54.178 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:54.178 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:54.178 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:54.179 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:54.179 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:54.179 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:54.179 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:54.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.178+0000 7f0d85579640 10 monclient: get_monmap_and_config success 2024-03-27T03:08:54.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.178+0000 7f0d85579640 4 set_mon_vals no callback set 2024-03-27T03:08:54.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.179+0000 7f0d85579640 10 monclient: shutdown 2024-03-27T03:08:54.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.179+0000 7f0d85579640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:54.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.179+0000 7f0d85579640 1 -- 172.21.15.153:0/3559200076 >> v1:172.21.15.196:6789/0 conn(0x7f0d80142690 legacy=0x7f0d800b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:54.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.180+0000 7f0d85579640 1 -- 172.21.15.153:0/3559200076 shutdown_connections 2024-03-27T03:08:54.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.180+0000 7f0d85579640 1 -- 172.21.15.153:0/3559200076 >> 172.21.15.153:0/3559200076 conn(0x7f0d80066a00 msgr2=0x7f0d80066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:54.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.180+0000 7f0d85579640 1 -- 172.21.15.153:0/3559200076 shutdown_connections 2024-03-27T03:08:54.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.180+0000 7f0d85579640 1 -- 172.21.15.153:0/3559200076 wait complete. 2024-03-27T03:08:54.186 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.181+0000 7f0d85579640 10 monclient: build_initial_monmap 2024-03-27T03:08:54.186 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.181+0000 7f0d85579640 10 monclient: monmap: 2024-03-27T03:08:54.186 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:08:54.186 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:08:54.186 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:08:54.186 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:08:54.186 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:08:54.186 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:54.186 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.182+0000 7f0d85579640 1 librados: starting msgr at 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.182+0000 7f0d85579640 1 librados: starting objecter 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.182+0000 7f0d85579640 1 Processor -- start 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.182+0000 7f0d85579640 1 -- start start 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.182+0000 7f0d85579640 1 librados: setting wanted keys 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.182+0000 7f0d85579640 1 librados: calling monclient init 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.182+0000 7f0d85579640 10 monclient: init 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.184+0000 7f0d85579640 10 monclient: _reopen_session rank -1 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.184+0000 7f0d85579640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.184+0000 7f0d85579640 10 monclient(hunting): picked mon.noname-a con 0x7f0d800b17d0 addr v1:172.21.15.153:6789/0 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.184+0000 7f0d85579640 10 monclient(hunting): picked mon.noname-c con 0x7f0d801422b0 addr v1:172.21.15.153:6790/0 2024-03-27T03:08:54.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.184+0000 7f0d85579640 10 monclient(hunting): picked mon.noname-b con 0x7f0d80142690 addr v1:172.21.15.196:6789/0 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.184+0000 7f0d85579640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d800ac4c0 con 0x7f0d800b17d0 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.184+0000 7f0d7e7fc640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f0d801422b0 0x7f0d80130340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:34620/0 (socket says 172.21.15.153:34620) 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.184+0000 7f0d7e7fc640 1 -- 172.21.15.153:0/2443244067 learned_addr learned my addr 172.21.15.153:0/2443244067 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.184+0000 7f0d85579640 1 -- 172.21.15.153:0/2443244067 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d800ac7a0 con 0x7f0d80142690 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d85579640 1 -- 172.21.15.153:0/2443244067 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d800aca80 con 0x7f0d801422b0 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d85579640 10 monclient(hunting): _renew_subs 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d85579640 10 monclient(hunting): authenticate will time out at 1494.491873s 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 client.?.objecter ms_handle_connect 0x7f0d800b17d0 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 client.?.objecter ms_handle_connect 0x7f0d801422b0 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 client.?.objecter ms_handle_connect 0x7f0d80142690 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:08:54.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342691891 0 0) 0x7f0d800ac4c0 con 0x7f0d800b17d0 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 monclient(hunting): _negotiate 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 monclient(hunting): my global_id is 4347 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d60004eb0 con 0x7f0d800b17d0 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212474681 0 0) 0x7f0d800ac7a0 con 0x7f0d80142690 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 monclient(hunting): _negotiate 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 10 monclient(hunting): my global_id is 4354 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.185+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d800ac4c0 con 0x7f0d80142690 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031844458 0 0) 0x7f0d800aca80 con 0x7f0d801422b0 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 10 monclient(hunting): _negotiate 2024-03-27T03:08:54.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 10 monclient(hunting): my global_id is 4367 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d800ac7a0 con 0x7f0d801422b0 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2621463766 0 0) 0x7f0d800ac4c0 con 0x7f0d80142690 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d800aca80 con 0x7f0d80142690 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0d700018b0 con 0x7f0d80142690 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 10 monclient(hunting): dump: 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:54.190 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 523847361 0 0) 0x7f0d800ac7a0 con 0x7f0d801422b0 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d60006bb0 con 0x7f0d801422b0 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0d78001ec0 con 0x7f0d801422b0 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.186+0000 7f0d777fe640 10 monclient(hunting): dump: 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:54.191 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3202397393 0 0) 0x7f0d800aca80 con 0x7f0d80142690 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 >> v1:172.21.15.153:6790/0 conn(0x7f0d801422b0 legacy=0x7f0d80130340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 >> v1:172.21.15.153:6789/0 conn(0x7f0d800b17d0 legacy=0x7f0d8012fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d777fe640 10 monclient: _finish_hunting 0 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d777fe640 1 monclient: found mon.b 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d777fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d800acd60 con 0x7f0d80142690 2024-03-27T03:08:54.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d777fe640 10 monclient: _finish_auth 0 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d777fe640 10 monclient: _check_auth_tickets 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d85579640 5 monclient: authenticate success, global_id 4354 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d85579640 10 monclient: _renew_subs 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d85579640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.187+0000 7f0d85579640 1 -- 172.21.15.153:0/2443244067 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d80132780 con 0x7f0d80142690 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.188+0000 7f0d85579640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.188+0000 7f0d85579640 10 monclient: _renew_subs 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.188+0000 7f0d85579640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.188+0000 7f0d85579640 1 -- 172.21.15.153:0/2443244067 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d800ab7b0 con 0x7f0d80142690 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.188+0000 7f0d85579640 1 librados: init done 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.188+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d700035f0 con 0x7f0d80142690 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.188+0000 7f0d777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.188+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0d70001d80 con 0x7f0d80142690 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.188+0000 7f0d777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:08:54.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.188+0000 7f0d777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.188+0000 7f0d777fe640 10 monclient: dump: 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.188+0000 7f0d7d7fa640 4 set_mon_vals no callback set 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.190+0000 7f0d85579640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.190+0000 7f0d85579640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.190+0000 7f0d85579640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:54.194 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.190+0000 7f0d85579640 1 -- 172.21.15.153:0/2443244067 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d64005080 con 0x7f0d80142690 2024-03-27T03:08:54.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.191+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f0d70019770 con 0x7f0d80142690 2024-03-27T03:08:54.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.191+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f0d700837c0 con 0x7f0d80142690 2024-03-27T03:08:54.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.191+0000 7f0d777fe640 10 client.4354.objecter ms_dispatch 0x7f0d80000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:08:54.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.191+0000 7f0d777fe640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:08:54.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.191+0000 7f0d777fe640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:08:54.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.191+0000 7f0d777fe640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-27T03:08:54.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.191+0000 7f0d7effd640 10 monclient: get_auth_request con 0x7f0d6006e160 auth_method 0 2024-03-27T03:08:54.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.192+0000 7f0d777fe640 10 client.4354.objecter ms_handle_connect 0x7f0d6006e160 2024-03-27T03:08:54.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.197+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d7004f100 con 0x7f0d80142690 2024-03-27T03:08:54.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.197+0000 7f0d777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:08:54.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.197+0000 7f0d777fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:08:54.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.552+0000 7f0d85579640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:54.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.552+0000 7f0d85579640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:54.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.552+0000 7f0d85579640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:08:54.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.552+0000 7f0d85579640 1 -- 172.21.15.153:0/2443244067 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d64005330 con 0x7f0d80142690 2024-03-27T03:08:54.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.553+0000 7f0d777fe640 1 -- 172.21.15.153:0/2443244067 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f0d7004f100 con 0x7f0d80142690 2024-03-27T03:08:54.548 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:08:54.548 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:08:54.549 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.554+0000 7f0d777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:08:54.549 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.554+0000 7f0d777fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:08:54.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.558+0000 7f0d85579640 10 librados: watch_flush enter 2024-03-27T03:08:54.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.559+0000 7f0d85579640 10 librados: watch_flush exit 2024-03-27T03:08:54.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.559+0000 7f0d85579640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-27T03:08:54.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.559+0000 7f0d85579640 10 client.4354.objecter successfully canceled tick 2024-03-27T03:08:54.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.559+0000 7f0d85579640 1 -- 172.21.15.153:0/2443244067 >> v1:172.21.15.153:6816/2637937781 conn(0x7f0d6006e160 legacy=0x7f0d60070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:54.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.559+0000 7f0d85579640 10 monclient: shutdown 2024-03-27T03:08:54.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.559+0000 7f0d85579640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:08:54.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.559+0000 7f0d85579640 1 -- 172.21.15.153:0/2443244067 >> v1:172.21.15.196:6789/0 conn(0x7f0d80142690 legacy=0x7f0d801e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:08:54.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.559+0000 7f0d85579640 1 -- 172.21.15.153:0/2443244067 shutdown_connections 2024-03-27T03:08:54.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.559+0000 7f0d85579640 1 -- 172.21.15.153:0/2443244067 >> 172.21.15.153:0/2443244067 conn(0x7f0d80066a00 msgr2=0x7f0d8013c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:08:54.554 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.559+0000 7f0d85579640 1 -- 172.21.15.153:0/2443244067 shutdown_connections 2024-03-27T03:08:54.554 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.559+0000 7f0d85579640 1 -- 172.21.15.153:0/2443244067 wait complete. 2024-03-27T03:08:54.554 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:08:54.559+0000 7f0d85579640 1 librados: shutdown 2024-03-27T03:08:54.569 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:09:01.376 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:09:01.531 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.538+0000 7f0dc1f02640 10 monclient: get_monmap_and_config 2024-03-27T03:09:01.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.538+0000 7f0dc1f02640 10 monclient: build_initial_monmap 2024-03-27T03:09:01.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.538+0000 7f0dc1f02640 10 monclient: monmap: 2024-03-27T03:09:01.532 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:01.532 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:01.532 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:09:01.539634+0000 2024-03-27T03:09:01.532 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:09:01.539634+0000 2024-03-27T03:09:01.532 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:01.532 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:01.532 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:01.532 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:09:01.532 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:09:01.532 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:01.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.539+0000 7f0dc1f02640 1 Processor -- start 2024-03-27T03:09:01.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.539+0000 7f0dc1f02640 1 -- start start 2024-03-27T03:09:01.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.539+0000 7f0dc1f02640 10 monclient: init 2024-03-27T03:09:01.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.540+0000 7f0dc1f02640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:01.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.540+0000 7f0dc1f02640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:09:01.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.540+0000 7f0dc1f02640 10 monclient(hunting): picked mon.noname-c con 0x7f0dbc0b4710 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:01.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.540+0000 7f0dc1f02640 10 monclient(hunting): picked mon.noname-b con 0x7f0dbc0b3970 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:01.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.540+0000 7f0dc1f02640 10 monclient(hunting): picked mon.noname-a con 0x7f0dbc13f390 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:01.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.540+0000 7f0dc1f02640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc0af020 con 0x7f0dbc13f390 2024-03-27T03:09:01.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.540+0000 7f0dc1f02640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc0b0910 con 0x7f0dbc0b4710 2024-03-27T03:09:01.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.540+0000 7f0dc1f02640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc0b0bf0 con 0x7f0dbc0b3970 2024-03-27T03:09:01.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.540+0000 7f0dc1f02640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:01.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.540+0000 7f0dc1f02640 10 monclient(hunting): authenticate will time out at 1501.846477s 2024-03-27T03:09:01.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dbaffd640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f0dbc0b3970 0x7f0dbc0af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:39398/0 (socket says 172.21.15.153:39398) 2024-03-27T03:09:01.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dbaffd640 1 -- 172.21.15.153:0/4283585679 learned_addr learned my addr 172.21.15.153:0/4283585679 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:01.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590958640 0 0) 0x7f0dbc0b0bf0 con 0x7f0dbc0b3970 2024-03-27T03:09:01.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dba7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:09:01.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:01.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:01.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dba7fc640 10 monclient(hunting): my global_id is 4370 2024-03-27T03:09:01.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da4003520 con 0x7f0dbc0b3970 2024-03-27T03:09:01.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334837562 0 0) 0x7f0dbc0b0910 con 0x7f0dbc0b4710 2024-03-27T03:09:01.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dba7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:09:01.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:01.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:01.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dba7fc640 10 monclient(hunting): my global_id is 4357 2024-03-27T03:09:01.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.541+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dbc0b0bf0 con 0x7f0dbc0b4710 2024-03-27T03:09:01.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470002023 0 0) 0x7f0dbc0af020 con 0x7f0dbc13f390 2024-03-27T03:09:01.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:09:01.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient(hunting): my global_id is 4350 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dbc0b0910 con 0x7f0dbc13f390 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1672254669 0 0) 0x7f0da4003520 con 0x7f0dbc0b3970 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 >> v1:172.21.15.196:6789/0 conn(0x7f0dbc0b4710 legacy=0x7f0dbc0b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 >> v1:172.21.15.153:6789/0 conn(0x7f0dbc13f390 legacy=0x7f0dbc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 1 monclient: found mon.noname-b 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc0aeb00 con 0x7f0dbc0b3970 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient: _finish_auth 0 2024-03-27T03:09:01.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0db0001610 con 0x7f0dbc0b3970 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient: dump: 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:01.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dc1f02640 5 monclient: authenticate success, global_id 4370 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dc1f02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0db0001770 con 0x7f0dbc0b3970 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 1 -- 172.21.15.153:0/4283585679 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0db0001bc0 con 0x7f0dbc0b3970 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dba7fc640 10 monclient: dump: 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:01.539 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:01.540 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:01.540 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:01.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dc1f02640 10 monclient: get_monmap_and_config success 2024-03-27T03:09:01.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.542+0000 7f0dc1f02640 4 set_mon_vals no callback set 2024-03-27T03:09:01.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.543+0000 7f0dc1f02640 10 monclient: shutdown 2024-03-27T03:09:01.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.543+0000 7f0dc1f02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:01.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.543+0000 7f0dc1f02640 1 -- 172.21.15.153:0/4283585679 >> v1:172.21.15.153:6790/0 conn(0x7f0dbc0b3970 legacy=0x7f0dbc0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:01.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.543+0000 7f0dc1f02640 1 -- 172.21.15.153:0/4283585679 shutdown_connections 2024-03-27T03:09:01.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.543+0000 7f0dc1f02640 1 -- 172.21.15.153:0/4283585679 >> 172.21.15.153:0/4283585679 conn(0x7f0dbc066a00 msgr2=0x7f0dbc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:01.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.543+0000 7f0dc1f02640 1 -- 172.21.15.153:0/4283585679 shutdown_connections 2024-03-27T03:09:01.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.543+0000 7f0dc1f02640 1 -- 172.21.15.153:0/4283585679 wait complete. 2024-03-27T03:09:01.545 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.544+0000 7f0dc1f02640 10 monclient: build_initial_monmap 2024-03-27T03:09:01.545 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.544+0000 7f0dc1f02640 10 monclient: monmap: 2024-03-27T03:09:01.545 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:01.545 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.544+0000 7f0dc1f02640 1 librados: starting msgr at 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.544+0000 7f0dc1f02640 1 librados: starting objecter 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.545+0000 7f0dc1f02640 1 Processor -- start 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.545+0000 7f0dc1f02640 1 -- start start 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.545+0000 7f0dc1f02640 1 librados: setting wanted keys 2024-03-27T03:09:01.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.545+0000 7f0dc1f02640 1 librados: calling monclient init 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.545+0000 7f0dc1f02640 10 monclient: init 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0dc1f02640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0dc1f02640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0dc1f02640 10 monclient(hunting): picked mon.noname-b con 0x7f0dbc0b3970 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0dc1f02640 10 monclient(hunting): picked mon.noname-a con 0x7f0dbc0b4710 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0dc1f02640 10 monclient(hunting): picked mon.noname-c con 0x7f0dbc13f390 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0dc1f02640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc140610 con 0x7f0dbc0b4710 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0dc1f02640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc1d3930 con 0x7f0dbc0b3970 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0dc1f02640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc13ff70 con 0x7f0dbc13f390 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0dc1f02640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0dc1f02640 10 monclient(hunting): authenticate will time out at 1501.851881s 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0dbbfff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f0dbc13f390 0x7f0dbc1de240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:39614/0 (socket says 172.21.15.153:39614) 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0dbbfff640 1 -- 172.21.15.153:0/1710997604 learned_addr learned my addr 172.21.15.153:0/1710997604 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0da3fff640 10 client.?.objecter ms_handle_connect 0x7f0dbc0b4710 2024-03-27T03:09:01.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0da3fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0da3fff640 10 client.?.objecter ms_handle_connect 0x7f0dbc0b3970 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.546+0000 7f0da3fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677119158 0 0) 0x7f0dbc140610 con 0x7f0dbc0b4710 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): _negotiate 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): my global_id is 4353 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da8004eb0 con 0x7f0dbc0b4710 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 client.?.objecter ms_handle_connect 0x7f0dbc13f390 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2835753226 0 0) 0x7f0da8004eb0 con 0x7f0dbc0b4710 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dbc140610 con 0x7f0dbc0b4710 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0db0001a80 con 0x7f0dbc0b4710 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:01.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): dump: 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458439828 0 0) 0x7f0dbc13ff70 con 0x7f0dbc13f390 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): _negotiate 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:01.549 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): my global_id is 4373 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da8004eb0 con 0x7f0dbc13f390 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989791774 0 0) 0x7f0dbc1d3930 con 0x7f0dbc0b3970 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): _negotiate 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 10 monclient(hunting): my global_id is 4360 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.547+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dbc13ff70 con 0x7f0dbc0b3970 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3929690729 0 0) 0x7f0da8004eb0 con 0x7f0dbc13f390 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dbc1d3930 con 0x7f0dbc13f390 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0db4001b30 con 0x7f0dbc13f390 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 10 monclient(hunting): dump: 2024-03-27T03:09:01.550 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3452734908 0 0) 0x7f0dbc140610 con 0x7f0dbc0b4710 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 >> v1:172.21.15.153:6790/0 conn(0x7f0dbc13f390 legacy=0x7f0dbc1de240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 >> v1:172.21.15.196:6789/0 conn(0x7f0dbc0b3970 legacy=0x7f0dbc1d3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 10 monclient: _finish_hunting 0 2024-03-27T03:09:01.551 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 1 monclient: found mon.a 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc1d4560 con 0x7f0dbc0b4710 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 10 monclient: _finish_auth 0 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 10 monclient: _check_auth_tickets 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0da3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0dc1f02640 5 monclient: authenticate success, global_id 4353 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0dc1f02640 10 monclient: _renew_subs 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0dc1f02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0dc1f02640 1 -- 172.21.15.153:0/1710997604 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dbc13a1f0 con 0x7f0dbc0b4710 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0dc1f02640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0dc1f02640 10 monclient: _renew_subs 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0dc1f02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0dc1f02640 1 -- 172.21.15.153:0/1710997604 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dbc070890 con 0x7f0dbc0b4710 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.548+0000 7f0dc1f02640 1 librados: init done 2024-03-27T03:09:01.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.549+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0db0001be0 con 0x7f0dbc0b4710 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.549+0000 7f0da3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.549+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f0db0004c30 con 0x7f0dbc0b4710 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.549+0000 7f0da3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.549+0000 7f0db9ffb640 4 set_mon_vals no callback set 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.549+0000 7f0da3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.550+0000 7f0da3fff640 10 monclient: dump: 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:01.553 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:01.554 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:01.554 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.550+0000 7f0dc1f02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:01.554 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.550+0000 7f0dc1f02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:01.554 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.550+0000 7f0dc1f02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:01.554 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.550+0000 7f0dc1f02640 1 -- 172.21.15.153:0/1710997604 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9c005080 con 0x7f0dbc0b4710 2024-03-27T03:09:01.554 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.552+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f0db001ac00 con 0x7f0dbc0b4710 2024-03-27T03:09:01.555 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.553+0000 7f0dbb7fe640 10 monclient: get_auth_request con 0x7f0da806e000 auth_method 0 2024-03-27T03:09:01.555 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.553+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f0db0084a70 con 0x7f0dbc0b4710 2024-03-27T03:09:01.555 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.553+0000 7f0da3fff640 10 client.4353.objecter ms_dispatch 0x7f0dbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:09:01.555 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.553+0000 7f0da3fff640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:09:01.555 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.553+0000 7f0da3fff640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:09:01.555 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.553+0000 7f0da3fff640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-27T03:09:01.555 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.553+0000 7f0da3fff640 10 client.4353.objecter ms_handle_connect 0x7f0da806e000 2024-03-27T03:09:01.556 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.559+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0db0050350 con 0x7f0dbc0b4710 2024-03-27T03:09:01.556 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.559+0000 7f0da3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:01.556 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.559+0000 7f0da3fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:09:01.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.931+0000 7f0dc1f02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:01.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.931+0000 7f0dc1f02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:01.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.931+0000 7f0dc1f02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:01.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.931+0000 7f0dc1f02640 1 -- 172.21.15.153:0/1710997604 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d9c005330 con 0x7f0dbc0b4710 2024-03-27T03:09:01.925 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.932+0000 7f0da3fff640 1 -- 172.21.15.153:0/1710997604 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f0db0052280 con 0x7f0dbc0b4710 2024-03-27T03:09:01.925 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.932+0000 7f0da3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:01.925 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.932+0000 7f0da3fff640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:09:01.925 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:09:01.925 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:09:01.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.935+0000 7f0dc1f02640 10 librados: watch_flush enter 2024-03-27T03:09:01.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.935+0000 7f0dc1f02640 10 librados: watch_flush exit 2024-03-27T03:09:01.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.935+0000 7f0dc1f02640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-27T03:09:01.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.935+0000 7f0dc1f02640 10 client.4353.objecter successfully canceled tick 2024-03-27T03:09:01.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.935+0000 7f0dc1f02640 1 -- 172.21.15.153:0/1710997604 >> v1:172.21.15.153:6816/2637937781 conn(0x7f0da806e000 legacy=0x7f0da80704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:01.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.935+0000 7f0dc1f02640 10 monclient: shutdown 2024-03-27T03:09:01.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.936+0000 7f0dc1f02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:01.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.936+0000 7f0dc1f02640 1 -- 172.21.15.153:0/1710997604 >> v1:172.21.15.153:6789/0 conn(0x7f0dbc0b4710 legacy=0x7f0dbc06ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:01.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.936+0000 7f0dc1f02640 1 -- 172.21.15.153:0/1710997604 shutdown_connections 2024-03-27T03:09:01.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.936+0000 7f0dc1f02640 1 -- 172.21.15.153:0/1710997604 >> 172.21.15.153:0/1710997604 conn(0x7f0dbc066a00 msgr2=0x7f0dbc13f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:01.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.936+0000 7f0dc1f02640 1 -- 172.21.15.153:0/1710997604 shutdown_connections 2024-03-27T03:09:01.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.936+0000 7f0dc1f02640 1 -- 172.21.15.153:0/1710997604 wait complete. 2024-03-27T03:09:01.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:01.936+0000 7f0dc1f02640 1 librados: shutdown 2024-03-27T03:09:01.945 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:09:08.753 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:09:08.909 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.915+0000 7fb97c55d640 10 monclient: get_monmap_and_config 2024-03-27T03:09:08.909 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.915+0000 7fb97c55d640 10 monclient: build_initial_monmap 2024-03-27T03:09:08.909 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.916+0000 7fb97c55d640 10 monclient: monmap: 2024-03-27T03:09:08.909 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:08.909 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:08.909 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:09:08.916921+0000 2024-03-27T03:09:08.909 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:09:08.916921+0000 2024-03-27T03:09:08.909 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:08.909 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:08.910 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:08.910 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:09:08.910 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:09:08.910 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:08.915 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.919+0000 7fb97c55d640 1 Processor -- start 2024-03-27T03:09:08.915 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.921+0000 7fb97c55d640 1 -- start start 2024-03-27T03:09:08.915 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.921+0000 7fb97c55d640 10 monclient: init 2024-03-27T03:09:08.915 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.922+0000 7fb97c55d640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:08.915 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.922+0000 7fb97c55d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:09:08.915 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.922+0000 7fb97c55d640 10 monclient(hunting): picked mon.noname-c con 0x7fb9740b0860 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:08.916 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.922+0000 7fb97c55d640 10 monclient(hunting): picked mon.noname-a con 0x7fb97413ef30 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:08.916 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.922+0000 7fb97c55d640 10 monclient(hunting): picked mon.noname-b con 0x7fb9740af8e0 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:08.916 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.922+0000 7fb97c55d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9740b0c40 con 0x7fb97413ef30 2024-03-27T03:09:08.916 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.922+0000 7fb97c55d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9740aed10 con 0x7fb9740b0860 2024-03-27T03:09:08.916 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.922+0000 7fb97c55d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9740afd50 con 0x7fb9740af8e0 2024-03-27T03:09:08.916 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.922+0000 7fb97c55d640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:08.916 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.922+0000 7fb97c55d640 10 monclient(hunting): authenticate will time out at 1509.227970s 2024-03-27T03:09:08.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.929+0000 7fb97a2f9640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fb9740b0860 0x7fb97413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:52853/0 (socket says 172.21.15.153:52853) 2024-03-27T03:09:08.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.929+0000 7fb97a2f9640 1 -- 172.21.15.153:0/2051743270 learned_addr learned my addr 172.21.15.153:0/2051743270 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:08.926 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.931+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645575534 0 0) 0x7fb9740aed10 con 0x7fb9740b0860 2024-03-27T03:09:08.926 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.931+0000 7fb9792f7640 10 monclient(hunting): _negotiate 2024-03-27T03:09:08.926 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.931+0000 7fb9792f7640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:08.926 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.931+0000 7fb9792f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.931+0000 7fb9792f7640 10 monclient(hunting): my global_id is 4363 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.931+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb960003520 con 0x7fb9740b0860 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033065367 0 0) 0x7fb9740afd50 con 0x7fb9740af8e0 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient(hunting): _negotiate 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient(hunting): my global_id is 4376 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9740aed10 con 0x7fb9740af8e0 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578608276 0 0) 0x7fb9740b0c40 con 0x7fb97413ef30 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient(hunting): _negotiate 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient(hunting): my global_id is 4356 2024-03-27T03:09:08.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9740afd50 con 0x7fb97413ef30 2024-03-27T03:09:08.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2874929104 0 0) 0x7fb960003520 con 0x7fb9740b0860 2024-03-27T03:09:08.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 >> v1:172.21.15.153:6790/0 conn(0x7fb9740af8e0 legacy=0x7fb974149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:08.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 >> v1:172.21.15.153:6789/0 conn(0x7fb97413ef30 legacy=0x7fb9740af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:08.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient: _finish_hunting 0 2024-03-27T03:09:08.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 1 monclient: found mon.noname-c 2024-03-27T03:09:08.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:08.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:09:08.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9740ab1b0 con 0x7fb9740b0860 2024-03-27T03:09:08.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient: _finish_auth 0 2024-03-27T03:09:08.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient: _check_auth_tickets 2024-03-27T03:09:08.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:08.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fb96c001a10 con 0x7fb9740b0860 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient: dump: 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb96c001b70 con 0x7fb9740b0860 2024-03-27T03:09:08.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 1 -- 172.21.15.153:0/2051743270 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fb96c001fc0 con 0x7fb9740b0860 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb9792f7640 10 monclient: dump: 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:08.930 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:08.931 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb97c55d640 5 monclient: authenticate success, global_id 4363 2024-03-27T03:09:08.931 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb97c55d640 10 monclient: get_monmap_and_config success 2024-03-27T03:09:08.931 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.932+0000 7fb97c55d640 4 set_mon_vals no callback set 2024-03-27T03:09:08.931 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.933+0000 7fb97c55d640 10 monclient: shutdown 2024-03-27T03:09:08.931 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.933+0000 7fb97c55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:08.931 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.933+0000 7fb97c55d640 1 -- 172.21.15.153:0/2051743270 >> v1:172.21.15.196:6789/0 conn(0x7fb9740b0860 legacy=0x7fb97413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:08.931 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.934+0000 7fb97c55d640 1 -- 172.21.15.153:0/2051743270 shutdown_connections 2024-03-27T03:09:08.932 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.934+0000 7fb97c55d640 1 -- 172.21.15.153:0/2051743270 >> 172.21.15.153:0/2051743270 conn(0x7fb97406b660 msgr2=0x7fb97406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:08.932 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.937+0000 7fb97c55d640 1 -- 172.21.15.153:0/2051743270 shutdown_connections 2024-03-27T03:09:08.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.939+0000 7fb97c55d640 1 -- 172.21.15.153:0/2051743270 wait complete. 2024-03-27T03:09:08.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.940+0000 7fb97c55d640 10 monclient: build_initial_monmap 2024-03-27T03:09:08.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.940+0000 7fb97c55d640 10 monclient: monmap: 2024-03-27T03:09:08.933 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:08.934 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:09:08.934 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:09:08.934 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:09:08.934 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:08.934 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:08.934 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:08.934 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:09:08.934 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:09:08.934 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:08.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.940+0000 7fb97c55d640 1 librados: starting msgr at 2024-03-27T03:09:08.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.940+0000 7fb97c55d640 1 librados: starting objecter 2024-03-27T03:09:08.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.940+0000 7fb97c55d640 1 Processor -- start 2024-03-27T03:09:08.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.940+0000 7fb97c55d640 1 -- start start 2024-03-27T03:09:08.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.940+0000 7fb97c55d640 1 librados: setting wanted keys 2024-03-27T03:09:08.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.940+0000 7fb97c55d640 1 librados: calling monclient init 2024-03-27T03:09:08.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.940+0000 7fb97c55d640 10 monclient: init 2024-03-27T03:09:08.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.941+0000 7fb97c55d640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:08.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.941+0000 7fb97c55d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:09:08.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.941+0000 7fb97c55d640 10 monclient(hunting): picked mon.noname-a con 0x7fb9740af8e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:08.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.941+0000 7fb97c55d640 10 monclient(hunting): picked mon.noname-b con 0x7fb9740b0860 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:08.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.941+0000 7fb97c55d640 10 monclient(hunting): picked mon.noname-c con 0x7fb97413ef30 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:08.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.941+0000 7fb97c55d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb974071440 con 0x7fb9740af8e0 2024-03-27T03:09:08.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.941+0000 7fb97c55d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb974149900 con 0x7fb9740b0860 2024-03-27T03:09:08.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.941+0000 7fb97c55d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb974149be0 con 0x7fb97413ef30 2024-03-27T03:09:08.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.941+0000 7fb97c55d640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:08.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.941+0000 7fb97c55d640 10 monclient(hunting): authenticate will time out at 1509.247298s 2024-03-27T03:09:08.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.948+0000 7fb979af8640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fb9740b0860 0x7fb9741e30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:52999/0 (socket says 172.21.15.153:52999) 2024-03-27T03:09:08.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.949+0000 7fb979af8640 1 -- 172.21.15.153:0/512408295 learned_addr learned my addr 172.21.15.153:0/512408295 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:08.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.949+0000 7fb96affd640 10 client.?.objecter ms_handle_connect 0x7fb97413ef30 2024-03-27T03:09:08.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.949+0000 7fb96affd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:08.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.949+0000 7fb96affd640 10 client.?.objecter ms_handle_connect 0x7fb9740af8e0 2024-03-27T03:09:08.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.949+0000 7fb96affd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:08.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.949+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800913056 0 0) 0x7fb974149be0 con 0x7fb97413ef30 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): _negotiate 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): my global_id is 4379 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb958004eb0 con 0x7fb97413ef30 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918036816 0 0) 0x7fb974071440 con 0x7fb9740af8e0 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): _negotiate 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): my global_id is 4359 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb974149be0 con 0x7fb9740af8e0 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 client.?.objecter ms_handle_connect 0x7fb9740b0860 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3570485803 0 0) 0x7fb958004eb0 con 0x7fb97413ef30 2024-03-27T03:09:08.945 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb974071440 con 0x7fb97413ef30 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1019094977 0 0) 0x7fb974149be0 con 0x7fb9740af8e0 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb958004eb0 con 0x7fb9740af8e0 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517973670 0 0) 0x7fb974149900 con 0x7fb9740b0860 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): _negotiate 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): my global_id is 4366 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb974149be0 con 0x7fb9740b0860 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fb9700017e0 con 0x7fb97413ef30 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.950+0000 7fb96affd640 10 monclient(hunting): dump: 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:08.946 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fb96c001e80 con 0x7fb9740af8e0 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 10 monclient(hunting): dump: 2024-03-27T03:09:08.947 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 244772628 0 0) 0x7fb974071440 con 0x7fb97413ef30 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 >> v1:172.21.15.196:6789/0 conn(0x7fb9740b0860 legacy=0x7fb9741e30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 >> v1:172.21.15.153:6789/0 conn(0x7fb9740af8e0 legacy=0x7fb974148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 10 monclient: _finish_hunting 0 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 1 monclient: found mon.c 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:08.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb974071940 con 0x7fb97413ef30 2024-03-27T03:09:08.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 10 monclient: _finish_auth 0 2024-03-27T03:09:08.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 10 monclient: _check_auth_tickets 2024-03-27T03:09:08.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb96affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:08.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb97c55d640 5 monclient: authenticate success, global_id 4379 2024-03-27T03:09:08.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb97c55d640 10 monclient: _renew_subs 2024-03-27T03:09:08.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb97c55d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:08.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb97c55d640 1 -- 172.21.15.153:0/512408295 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb974071b30 con 0x7fb97413ef30 2024-03-27T03:09:08.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb97c55d640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:09:08.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb97c55d640 10 monclient: _renew_subs 2024-03-27T03:09:08.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb97c55d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:08.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb97c55d640 1 -- 172.21.15.153:0/512408295 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97414b400 con 0x7fb97413ef30 2024-03-27T03:09:08.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.951+0000 7fb97c55d640 1 librados: init done 2024-03-27T03:09:08.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.953+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb970001d10 con 0x7fb97413ef30 2024-03-27T03:09:08.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.953+0000 7fb96affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:08.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.953+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fb9700037b0 con 0x7fb97413ef30 2024-03-27T03:09:08.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.953+0000 7fb96affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:08.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.953+0000 7fb96affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:09:08.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.953+0000 7fb96affd640 10 monclient: dump: 2024-03-27T03:09:08.950 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.953+0000 7fb978af6640 4 set_mon_vals no callback set 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.954+0000 7fb97c55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.954+0000 7fb97c55d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.954+0000 7fb97c55d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:08.951 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.954+0000 7fb97c55d640 1 -- 172.21.15.153:0/512408295 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95c005080 con 0x7fb97413ef30 2024-03-27T03:09:08.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.957+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fb970019800 con 0x7fb97413ef30 2024-03-27T03:09:08.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.957+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fb970083750 con 0x7fb97413ef30 2024-03-27T03:09:08.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.958+0000 7fb96affd640 10 client.4379.objecter ms_dispatch 0x7fb974000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:09:08.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.958+0000 7fb96affd640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:09:08.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.958+0000 7fb96affd640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:09:08.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.958+0000 7fb96affd640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-27T03:09:08.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.959+0000 7fb97a2f9640 10 monclient: get_auth_request con 0x7fb95806df90 auth_method 0 2024-03-27T03:09:08.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.959+0000 7fb96affd640 10 client.4379.objecter ms_handle_connect 0x7fb95806df90 2024-03-27T03:09:08.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.963+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb97004ee60 con 0x7fb97413ef30 2024-03-27T03:09:08.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.963+0000 7fb96affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:08.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:08.963+0000 7fb96affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:09:09.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.294+0000 7fb97c55d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:09.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.294+0000 7fb97c55d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:09.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.294+0000 7fb97c55d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:09.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.294+0000 7fb97c55d640 1 -- 172.21.15.153:0/512408295 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb95c005330 con 0x7fb97413ef30 2024-03-27T03:09:09.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.298+0000 7fb96affd640 1 -- 172.21.15.153:0/512408295 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fb970050f60 con 0x7fb97413ef30 2024-03-27T03:09:09.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.298+0000 7fb96affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:09.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.298+0000 7fb96affd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:09:09.291 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:09:09.291 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:09:09.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.301+0000 7fb97c55d640 10 librados: watch_flush enter 2024-03-27T03:09:09.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.301+0000 7fb97c55d640 10 librados: watch_flush exit 2024-03-27T03:09:09.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.301+0000 7fb97c55d640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-27T03:09:09.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.301+0000 7fb97c55d640 10 client.4379.objecter successfully canceled tick 2024-03-27T03:09:09.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.302+0000 7fb97c55d640 1 -- 172.21.15.153:0/512408295 >> v1:172.21.15.153:6816/2637937781 conn(0x7fb95806df90 legacy=0x7fb958070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:09.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.302+0000 7fb97c55d640 10 monclient: shutdown 2024-03-27T03:09:09.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.302+0000 7fb97c55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:09.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.302+0000 7fb97c55d640 1 -- 172.21.15.153:0/512408295 >> v1:172.21.15.153:6790/0 conn(0x7fb97413ef30 legacy=0x7fb9741e67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:09.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.303+0000 7fb97c55d640 1 -- 172.21.15.153:0/512408295 shutdown_connections 2024-03-27T03:09:09.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.303+0000 7fb97c55d640 1 -- 172.21.15.153:0/512408295 >> 172.21.15.153:0/512408295 conn(0x7fb97406b660 msgr2=0x7fb97413dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:09.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.304+0000 7fb97c55d640 1 -- 172.21.15.153:0/512408295 shutdown_connections 2024-03-27T03:09:09.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.304+0000 7fb97c55d640 1 -- 172.21.15.153:0/512408295 wait complete. 2024-03-27T03:09:09.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:09.304+0000 7fb97c55d640 1 librados: shutdown 2024-03-27T03:09:09.316 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:09:16.122 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:09:16.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.277+0000 7f8263c4e640 10 monclient: get_monmap_and_config 2024-03-27T03:09:16.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.277+0000 7f8263c4e640 10 monclient: build_initial_monmap 2024-03-27T03:09:16.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.277+0000 7f8263c4e640 10 monclient: monmap: 2024-03-27T03:09:16.270 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:16.270 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:16.271 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:09:16.278356+0000 2024-03-27T03:09:16.271 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:09:16.278356+0000 2024-03-27T03:09:16.271 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:16.271 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:16.271 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:16.271 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:09:16.271 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:09:16.271 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:16.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.279+0000 7f8263c4e640 1 Processor -- start 2024-03-27T03:09:16.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.279+0000 7f8263c4e640 1 -- start start 2024-03-27T03:09:16.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.279+0000 7f8263c4e640 10 monclient: init 2024-03-27T03:09:16.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.280+0000 7f8263c4e640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:16.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.280+0000 7f8263c4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:09:16.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.280+0000 7f8263c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f825c0b4330 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:16.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.280+0000 7f8263c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f825c1470b0 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:16.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.280+0000 7f8263c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f825c1439b0 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:16.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.280+0000 7f8263c4e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c06f2a0 con 0x7f825c0b4330 2024-03-27T03:09:16.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.280+0000 7f8263c4e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c146b90 con 0x7f825c1439b0 2024-03-27T03:09:16.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.280+0000 7f8263c4e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c0b2ca0 con 0x7f825c1470b0 2024-03-27T03:09:16.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.280+0000 7f8263c4e640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:16.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.280+0000 7f8263c4e640 10 monclient(hunting): authenticate will time out at 1516.586249s 2024-03-27T03:09:16.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.281+0000 7f82621eb640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f825c1439b0 0x7f825c151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:59446/0 (socket says 172.21.15.153:59446) 2024-03-27T03:09:16.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.281+0000 7f82621eb640 1 -- 172.21.15.153:0/676566367 learned_addr learned my addr 172.21.15.153:0/676566367 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:16.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.283+0000 7f82609e8640 1 -- 172.21.15.153:0/676566367 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405613952 0 0) 0x7f825c146b90 con 0x7f825c1439b0 2024-03-27T03:09:16.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.283+0000 7f82609e8640 10 monclient(hunting): _negotiate 2024-03-27T03:09:16.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.283+0000 7f82609e8640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:16.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.283+0000 7f82609e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:16.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.283+0000 7f82609e8640 10 monclient(hunting): my global_id is 4369 2024-03-27T03:09:16.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.283+0000 7f82609e8640 1 -- 172.21.15.153:0/676566367 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8248003520 con 0x7f825c1439b0 2024-03-27T03:09:16.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.283+0000 7f82609e8640 1 -- 172.21.15.153:0/676566367 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288701314 0 0) 0x7f825c0b2ca0 con 0x7f825c1470b0 2024-03-27T03:09:16.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.283+0000 7f82609e8640 10 monclient(hunting): _negotiate 2024-03-27T03:09:16.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.283+0000 7f82609e8640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:16.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.283+0000 7f82609e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:16.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.283+0000 7f82609e8640 10 monclient(hunting): my global_id is 4382 2024-03-27T03:09:16.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.283+0000 7f82609e8640 1 -- 172.21.15.153:0/676566367 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f825c146b90 con 0x7f825c1470b0 2024-03-27T03:09:16.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.284+0000 7f82609e8640 1 -- 172.21.15.153:0/676566367 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 856901805 0 0) 0x7f825c146b90 con 0x7f825c1470b0 2024-03-27T03:09:16.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.284+0000 7f82609e8640 1 -- 172.21.15.153:0/676566367 >> v1:172.21.15.196:6789/0 conn(0x7f825c1439b0 legacy=0x7f825c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:16.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.284+0000 7f82609e8640 1 -- 172.21.15.153:0/676566367 >> v1:172.21.15.153:6789/0 conn(0x7f825c0b4330 legacy=0x7f825c0b4710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:09:16.279 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 10 monclient: _finish_hunting 0 2024-03-27T03:09:16.279 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 1 monclient: found mon.noname-b 2024-03-27T03:09:16.279 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:16.279 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:09:16.279 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 1 -- 172.21.15.153:0/676566367 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c06e0c0 con 0x7f825c1470b0 2024-03-27T03:09:16.279 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 10 monclient: _finish_auth 0 2024-03-27T03:09:16.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 10 monclient: _check_auth_tickets 2024-03-27T03:09:16.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:16.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 1 -- 172.21.15.153:0/676566367 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f824c001670 con 0x7f825c1470b0 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 10 monclient: dump: 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f8263c4e640 5 monclient: authenticate success, global_id 4382 2024-03-27T03:09:16.281 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f8263c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 1 -- 172.21.15.153:0/676566367 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f824c0017d0 con 0x7f825c1470b0 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 1 -- 172.21.15.153:0/676566367 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f824c001c20 con 0x7f825c1470b0 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f82609e8640 10 monclient: dump: 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:16.282 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:16.283 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:16.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f8263c4e640 10 monclient: get_monmap_and_config success 2024-03-27T03:09:16.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.285+0000 7f8263c4e640 4 set_mon_vals no callback set 2024-03-27T03:09:16.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.286+0000 7f8263c4e640 10 monclient: shutdown 2024-03-27T03:09:16.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.286+0000 7f8263c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:16.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.286+0000 7f8263c4e640 1 -- 172.21.15.153:0/676566367 >> v1:172.21.15.153:6790/0 conn(0x7f825c1470b0 legacy=0x7f825c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:16.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.286+0000 7f8263c4e640 1 -- 172.21.15.153:0/676566367 shutdown_connections 2024-03-27T03:09:16.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.286+0000 7f8263c4e640 1 -- 172.21.15.153:0/676566367 >> 172.21.15.153:0/676566367 conn(0x7f825c066a00 msgr2=0x7f825c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:16.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.287+0000 7f8263c4e640 1 -- 172.21.15.153:0/676566367 shutdown_connections 2024-03-27T03:09:16.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.287+0000 7f8263c4e640 1 -- 172.21.15.153:0/676566367 wait complete. 2024-03-27T03:09:16.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.288+0000 7f8263c4e640 10 monclient: build_initial_monmap 2024-03-27T03:09:16.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.288+0000 7f8263c4e640 10 monclient: monmap: 2024-03-27T03:09:16.284 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:16.284 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:09:16.284 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.289+0000 7f8263c4e640 1 librados: starting msgr at 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.289+0000 7f8263c4e640 1 librados: starting objecter 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.289+0000 7f8263c4e640 1 Processor -- start 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.289+0000 7f8263c4e640 1 -- start start 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.289+0000 7f8263c4e640 1 librados: setting wanted keys 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.289+0000 7f8263c4e640 1 librados: calling monclient init 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.289+0000 7f8263c4e640 10 monclient: init 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.290+0000 7f8263c4e640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:16.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.290+0000 7f8263c4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.290+0000 7f8263c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f825c0b4330 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.294+0000 7f8263c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f825c1439b0 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.294+0000 7f8263c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f825c1470b0 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.295+0000 7f82619ea640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f825c0b4330 0x7f825c1deec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:42329/0 (socket says 172.21.15.153:42329) 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.295+0000 7f82619ea640 1 -- 172.21.15.153:0/311135132 learned_addr learned my addr 172.21.15.153:0/311135132 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.295+0000 7f82527fc640 10 client.?.objecter ms_handle_connect 0x7f825c0b4330 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.295+0000 7f82527fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.295+0000 7f82527fc640 10 client.?.objecter ms_handle_connect 0x7f825c1470b0 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.295+0000 7f82527fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.295+0000 7f8263c4e640 1 -- 172.21.15.153:0/311135132 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c0718f0 con 0x7f825c0b4330 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f8263c4e640 1 -- 172.21.15.153:0/311135132 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c11f770 con 0x7f825c1470b0 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f8263c4e640 1 -- 172.21.15.153:0/311135132 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c1df5d0 con 0x7f825c1439b0 2024-03-27T03:09:16.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973140021 0 0) 0x7f825c0718f0 con 0x7f825c0b4330 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f8263c4e640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f8263c4e640 10 monclient(hunting): authenticate will time out at 1516.602023s 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f82527fc640 10 monclient(hunting): _negotiate 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f82527fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f82527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f82527fc640 10 monclient(hunting): my global_id is 4365 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8240004eb0 con 0x7f825c0b4330 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247163391 0 0) 0x7f825c11f770 con 0x7f825c1470b0 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f82527fc640 10 monclient(hunting): _negotiate 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f82527fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f82527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f82527fc640 10 monclient(hunting): my global_id is 4372 2024-03-27T03:09:16.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.296+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f825c0718f0 con 0x7f825c1470b0 2024-03-27T03:09:16.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.297+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2819221272 0 0) 0x7f8240004eb0 con 0x7f825c0b4330 2024-03-27T03:09:16.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.297+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f825c11f770 con 0x7f825c0b4330 2024-03-27T03:09:16.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.297+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8254001c10 con 0x7f825c0b4330 2024-03-27T03:09:16.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.297+0000 7f82527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:16.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.297+0000 7f82527fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:09:16.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.297+0000 7f82527fc640 10 monclient(hunting): dump: 2024-03-27T03:09:16.292 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:16.292 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:16.292 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:16.292 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:16.292 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:16.292 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:16.293 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:16.293 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:16.293 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:16.293 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:16.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.297+0000 7f82527fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:09:16.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.297+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2668349148 0 0) 0x7f825c11f770 con 0x7f825c0b4330 2024-03-27T03:09:16.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.297+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 >> v1:172.21.15.153:6790/0 conn(0x7f825c1439b0 legacy=0x7f825c11fc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:09:16.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.300+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 >> v1:172.21.15.196:6789/0 conn(0x7f825c1470b0 legacy=0x7f825c120380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:16.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.300+0000 7f82527fc640 10 monclient: _finish_hunting 0 2024-03-27T03:09:16.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.300+0000 7f82527fc640 1 monclient: found mon.a 2024-03-27T03:09:16.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.300+0000 7f82527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:16.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.300+0000 7f82527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:16.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.300+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c1df760 con 0x7f825c0b4330 2024-03-27T03:09:16.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.300+0000 7f82527fc640 10 monclient: _finish_auth 0 2024-03-27T03:09:16.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.300+0000 7f82527fc640 10 monclient: _check_auth_tickets 2024-03-27T03:09:16.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.300+0000 7f82527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:16.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.300+0000 7f82527fc640 10 client.?.objecter ms_handle_connect 0x7f825c1439b0 2024-03-27T03:09:16.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.300+0000 7f82527fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:16.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.301+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8254003990 con 0x7f825c0b4330 2024-03-27T03:09:16.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.301+0000 7f82527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:16.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.301+0000 7f8253fff640 4 set_mon_vals no callback set 2024-03-27T03:09:16.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f8263c4e640 5 monclient: authenticate success, global_id 4365 2024-03-27T03:09:16.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f8263c4e640 10 monclient: _renew_subs 2024-03-27T03:09:16.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f8263c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:16.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f8263c4e640 1 -- 172.21.15.153:0/311135132 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825c1df9b0 con 0x7f825c0b4330 2024-03-27T03:09:16.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f8263c4e640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:09:16.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f8263c4e640 10 monclient: _renew_subs 2024-03-27T03:09:16.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f8263c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f8263c4e640 1 -- 172.21.15.153:0/311135132 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f825c1e7380 con 0x7f825c0b4330 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f8263c4e640 1 librados: init done 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8254002100 con 0x7f825c0b4330 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f82527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f82527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.302+0000 7f82527fc640 10 monclient: dump: 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:16.298 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:16.299 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:16.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.304+0000 7f823ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:16.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.304+0000 7f823ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:16.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.304+0000 7f823ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:16.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.304+0000 7f823ffff640 1 -- 172.21.15.153:0/311135132 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f825c075f10 con 0x7f825c0b4330 2024-03-27T03:09:16.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.309+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f8254003de0 con 0x7f825c0b4330 2024-03-27T03:09:16.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.310+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f8254054ae0 con 0x7f825c0b4330 2024-03-27T03:09:16.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.310+0000 7f82527fc640 10 client.4365.objecter ms_dispatch 0x7f825c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:09:16.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.310+0000 7f82527fc640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:09:16.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.310+0000 7f82527fc640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:09:16.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.310+0000 7f82527fc640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-27T03:09:16.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.312+0000 7f82611e9640 10 monclient: get_auth_request con 0x7f824006da60 auth_method 0 2024-03-27T03:09:16.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.312+0000 7f82527fc640 10 client.4365.objecter ms_handle_connect 0x7f824006da60 2024-03-27T03:09:16.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.313+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8254050d80 con 0x7f825c0b4330 2024-03-27T03:09:16.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.313+0000 7f82527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:16.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.313+0000 7f82527fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:09:16.636 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.643+0000 7f823ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:16.636 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.643+0000 7f823ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:16.636 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.643+0000 7f823ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:16.636 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.643+0000 7f823ffff640 1 -- 172.21.15.153:0/311135132 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f825c144730 con 0x7f825c0b4330 2024-03-27T03:09:16.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.645+0000 7f82527fc640 1 -- 172.21.15.153:0/311135132 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f8254050d80 con 0x7f825c0b4330 2024-03-27T03:09:16.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.645+0000 7f82527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:16.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.645+0000 7f82527fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:09:16.638 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:09:16.638 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:09:16.641 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.648+0000 7f823ffff640 10 librados: watch_flush enter 2024-03-27T03:09:16.641 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.649+0000 7f823ffff640 10 librados: watch_flush exit 2024-03-27T03:09:16.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.649+0000 7f823ffff640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-27T03:09:16.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.649+0000 7f823ffff640 10 client.4365.objecter successfully canceled tick 2024-03-27T03:09:16.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.649+0000 7f823ffff640 1 -- 172.21.15.153:0/311135132 >> v1:172.21.15.153:6816/2637937781 conn(0x7f824006da60 legacy=0x7f824006ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:16.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.649+0000 7f823ffff640 10 monclient: shutdown 2024-03-27T03:09:16.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.649+0000 7f823ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:16.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.649+0000 7f823ffff640 1 -- 172.21.15.153:0/311135132 >> v1:172.21.15.153:6789/0 conn(0x7f825c0b4330 legacy=0x7f825c1deec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:16.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.649+0000 7f823ffff640 1 -- 172.21.15.153:0/311135132 shutdown_connections 2024-03-27T03:09:16.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.649+0000 7f823ffff640 1 -- 172.21.15.153:0/311135132 >> 172.21.15.153:0/311135132 conn(0x7f825c066a00 msgr2=0x7f825c1d1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:16.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.650+0000 7f823ffff640 1 -- 172.21.15.153:0/311135132 shutdown_connections 2024-03-27T03:09:16.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.651+0000 7f823ffff640 1 -- 172.21.15.153:0/311135132 wait complete. 2024-03-27T03:09:16.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:16.651+0000 7f823ffff640 1 librados: shutdown 2024-03-27T03:09:16.660 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:09:23.467 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:09:23.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.628+0000 7f191df66640 10 monclient: get_monmap_and_config 2024-03-27T03:09:23.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.628+0000 7f191df66640 10 monclient: build_initial_monmap 2024-03-27T03:09:23.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.628+0000 7f191df66640 10 monclient: monmap: 2024-03-27T03:09:23.623 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:23.623 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:23.623 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:09:23.629628+0000 2024-03-27T03:09:23.623 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:09:23.629628+0000 2024-03-27T03:09:23.624 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:23.624 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:23.624 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:23.624 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:09:23.624 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:09:23.624 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:23.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.630+0000 7f191df66640 1 Processor -- start 2024-03-27T03:09:23.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.630+0000 7f191df66640 1 -- start start 2024-03-27T03:09:23.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.630+0000 7f191df66640 10 monclient: init 2024-03-27T03:09:23.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.631+0000 7f191df66640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:23.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.631+0000 7f191df66640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:09:23.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.631+0000 7f191df66640 10 monclient(hunting): picked mon.noname-c con 0x7f1910006410 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:23.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.635+0000 7f191cf64640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f1910006410 0x7f19100345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:33451/0 (socket says 172.21.15.153:33451) 2024-03-27T03:09:23.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.636+0000 7f191cf64640 1 -- 172.21.15.153:0/85736811 learned_addr learned my addr 172.21.15.153:0/85736811 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:23.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.636+0000 7f191df66640 10 monclient(hunting): picked mon.noname-b con 0x7f191002a640 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:23.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.636+0000 7f191df66640 10 monclient(hunting): picked mon.noname-a con 0x7f191002b370 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:23.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.636+0000 7f191df66640 1 -- 172.21.15.153:0/85736811 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191002a120 con 0x7f191002b370 2024-03-27T03:09:23.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.636+0000 7f191df66640 1 -- 172.21.15.153:0/85736811 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19100c75b0 con 0x7f1910006410 2024-03-27T03:09:23.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.636+0000 7f191df66640 1 -- 172.21.15.153:0/85736811 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1910033110 con 0x7f191002a640 2024-03-27T03:09:23.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.636+0000 7f191df66640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:23.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.636+0000 7f191df66640 10 monclient(hunting): authenticate will time out at 1523.942396s 2024-03-27T03:09:23.631 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.637+0000 7f19177fe640 1 -- 172.21.15.153:0/85736811 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209352270 0 0) 0x7f19100c75b0 con 0x7f1910006410 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.637+0000 7f19177fe640 10 monclient(hunting): _negotiate 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.637+0000 7f19177fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.637+0000 7f19177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.637+0000 7f19177fe640 10 monclient(hunting): my global_id is 4375 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.637+0000 7f19177fe640 1 -- 172.21.15.153:0/85736811 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1908003520 con 0x7f1910006410 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.638+0000 7f19177fe640 1 -- 172.21.15.153:0/85736811 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236021372 0 0) 0x7f1910033110 con 0x7f191002a640 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.638+0000 7f19177fe640 10 monclient(hunting): _negotiate 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.638+0000 7f19177fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.638+0000 7f19177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.638+0000 7f19177fe640 10 monclient(hunting): my global_id is 4388 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.638+0000 7f19177fe640 1 -- 172.21.15.153:0/85736811 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19100c75b0 con 0x7f191002a640 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.638+0000 7f19177fe640 1 -- 172.21.15.153:0/85736811 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2966930916 0 0) 0x7f1908003520 con 0x7f1910006410 2024-03-27T03:09:23.632 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.638+0000 7f19177fe640 1 -- 172.21.15.153:0/85736811 >> v1:172.21.15.153:6790/0 conn(0x7f191002a640 legacy=0x7f19100c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:23.633 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.639+0000 7f19177fe640 1 -- 172.21.15.153:0/85736811 >> v1:172.21.15.153:6789/0 conn(0x7f191002b370 legacy=0x7f19100320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:09:23.639 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 10 monclient: _finish_hunting 0 2024-03-27T03:09:23.639 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 1 monclient: found mon.noname-c 2024-03-27T03:09:23.639 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:23.639 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:09:23.639 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 1 -- 172.21.15.153:0/85736811 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19100332a0 con 0x7f1910006410 2024-03-27T03:09:23.639 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 10 monclient: _finish_auth 0 2024-03-27T03:09:23.639 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 10 monclient: _check_auth_tickets 2024-03-27T03:09:23.639 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 1 -- 172.21.15.153:0/85736811 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f190c001690 con 0x7f1910006410 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 10 monclient: dump: 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f191df66640 5 monclient: authenticate success, global_id 4375 2024-03-27T03:09:23.640 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 1 -- 172.21.15.153:0/85736811 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f190c0017f0 con 0x7f1910006410 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f191df66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 1 -- 172.21.15.153:0/85736811 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f190c001b80 con 0x7f1910006410 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f19177fe640 10 monclient: dump: 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:23.641 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:23.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f191df66640 10 monclient: get_monmap_and_config success 2024-03-27T03:09:23.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.644+0000 7f191df66640 4 set_mon_vals no callback set 2024-03-27T03:09:23.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.645+0000 7f191df66640 10 monclient: shutdown 2024-03-27T03:09:23.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.646+0000 7f191df66640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:23.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.646+0000 7f191df66640 1 -- 172.21.15.153:0/85736811 >> v1:172.21.15.196:6789/0 conn(0x7f1910006410 legacy=0x7f19100345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:23.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.646+0000 7f191df66640 1 -- 172.21.15.153:0/85736811 shutdown_connections 2024-03-27T03:09:23.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.646+0000 7f191df66640 1 -- 172.21.15.153:0/85736811 >> 172.21.15.153:0/85736811 conn(0x7f19100058f0 msgr2=0x7f1910005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.647+0000 7f191df66640 1 -- 172.21.15.153:0/85736811 shutdown_connections 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.648+0000 7f191df66640 1 -- 172.21.15.153:0/85736811 wait complete. 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.649+0000 7f191df66640 10 monclient: build_initial_monmap 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.649+0000 7f191df66640 10 monclient: monmap: 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:09:23.643 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:09:23.644 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:23.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.649+0000 7f191df66640 1 librados: starting msgr at 2024-03-27T03:09:23.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.649+0000 7f191df66640 1 librados: starting objecter 2024-03-27T03:09:23.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.649+0000 7f191df66640 1 Processor -- start 2024-03-27T03:09:23.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.650+0000 7f191df66640 1 -- start start 2024-03-27T03:09:23.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.650+0000 7f191df66640 1 librados: setting wanted keys 2024-03-27T03:09:23.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.650+0000 7f191df66640 1 librados: calling monclient init 2024-03-27T03:09:23.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.650+0000 7f191df66640 10 monclient: init 2024-03-27T03:09:23.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.651+0000 7f191df66640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:23.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.651+0000 7f191df66640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:09:23.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.651+0000 7f191df66640 10 monclient(hunting): picked mon.noname-c con 0x7f1910006410 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:23.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.651+0000 7f191df66640 10 monclient(hunting): picked mon.noname-a con 0x7f191002a640 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:23.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.654+0000 7f191df66640 10 monclient(hunting): picked mon.noname-b con 0x7f191002b370 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:23.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.654+0000 7f191d765640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f191002b370 0x7f191002d3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:33577/0 (socket says 172.21.15.153:33577) 2024-03-27T03:09:23.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.654+0000 7f191d765640 1 -- 172.21.15.153:0/2455811721 learned_addr learned my addr 172.21.15.153:0/2455811721 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:23.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.654+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191002f740 con 0x7f191002a640 2024-03-27T03:09:23.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.654+0000 7f19157fa640 10 client.?.objecter ms_handle_connect 0x7f191002b370 2024-03-27T03:09:23.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.654+0000 7f19157fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:23.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.654+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191002fa20 con 0x7f191002b370 2024-03-27T03:09:23.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.655+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191015e580 con 0x7f1910006410 2024-03-27T03:09:23.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.655+0000 7f191df66640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:23.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.655+0000 7f191df66640 10 monclient(hunting): authenticate will time out at 1523.960695s 2024-03-27T03:09:23.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 10 client.?.objecter ms_handle_connect 0x7f1910006410 2024-03-27T03:09:23.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:23.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224463666 0 0) 0x7f191015e580 con 0x7f1910006410 2024-03-27T03:09:23.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 10 monclient(hunting): _negotiate 2024-03-27T03:09:23.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:23.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:23.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 10 monclient(hunting): my global_id is 4391 2024-03-27T03:09:23.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18fc004eb0 con 0x7f1910006410 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 10 client.?.objecter ms_handle_connect 0x7f191002a640 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417693250 0 0) 0x7f191002f740 con 0x7f191002a640 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 10 monclient(hunting): _negotiate 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 10 monclient(hunting): my global_id is 4368 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f191015e580 con 0x7f191002a640 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 927098016 0 0) 0x7f18fc004eb0 con 0x7f1910006410 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.657+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f191002f740 con 0x7f1910006410 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.658+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f190c001b00 con 0x7f1910006410 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.658+0000 7f19157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.658+0000 7f19157fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.658+0000 7f19157fa640 10 monclient(hunting): dump: 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:23.652 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:23.653 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:23.653 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:23.653 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:23.653 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:23.653 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:23.653 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:23.653 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:23.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.658+0000 7f19157fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:09:23.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.658+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293252860 0 0) 0x7f191002fa20 con 0x7f191002b370 2024-03-27T03:09:23.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.662+0000 7f19157fa640 10 monclient(hunting): _negotiate 2024-03-27T03:09:23.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.662+0000 7f19157fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:23.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.662+0000 7f19157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:23.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.662+0000 7f19157fa640 10 monclient(hunting): my global_id is 4378 2024-03-27T03:09:23.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.662+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18fc004eb0 con 0x7f191002b370 2024-03-27T03:09:23.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.662+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1397944612 0 0) 0x7f191015e580 con 0x7f191002a640 2024-03-27T03:09:23.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.662+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f191002fa20 con 0x7f191002a640 2024-03-27T03:09:23.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.663+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f19040018f0 con 0x7f191002a640 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.663+0000 7f19157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.663+0000 7f19157fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.663+0000 7f19157fa640 10 monclient(hunting): dump: 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:23.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.663+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 92887768 0 0) 0x7f191002f740 con 0x7f1910006410 2024-03-27T03:09:23.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.663+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 >> v1:172.21.15.196:6789/0 conn(0x7f191002b370 legacy=0x7f191002d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:23.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.663+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 >> v1:172.21.15.153:6789/0 conn(0x7f191002a640 legacy=0x7f1910031180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:23.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.663+0000 7f19157fa640 10 monclient: _finish_hunting 0 2024-03-27T03:09:23.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.663+0000 7f19157fa640 1 monclient: found mon.c 2024-03-27T03:09:23.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.663+0000 7f19157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:23.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.663+0000 7f19157fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:23.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.663+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19100d1c40 con 0x7f1910006410 2024-03-27T03:09:23.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.665+0000 7f19157fa640 10 monclient: _finish_auth 0 2024-03-27T03:09:23.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.665+0000 7f19157fa640 10 monclient: _check_auth_tickets 2024-03-27T03:09:23.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.665+0000 7f19157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:23.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.665+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f190c001c60 con 0x7f1910006410 2024-03-27T03:09:23.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.665+0000 7f19157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:23.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.665+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f190c004c30 con 0x7f1910006410 2024-03-27T03:09:23.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.665+0000 7f19157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:23.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.665+0000 7f19157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:09:23.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.665+0000 7f19157fa640 10 monclient: dump: 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.666+0000 7f191df66640 5 monclient: authenticate success, global_id 4391 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.666+0000 7f191df66640 10 monclient: _renew_subs 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.666+0000 7f191df66640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.666+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19100d1e90 con 0x7f1910006410 2024-03-27T03:09:23.662 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.666+0000 7f191df66640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:09:23.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.666+0000 7f191df66640 10 monclient: _renew_subs 2024-03-27T03:09:23.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.666+0000 7f191df66640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:23.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.666+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1910031a70 con 0x7f1910006410 2024-03-27T03:09:23.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.666+0000 7f191df66640 1 librados: init done 2024-03-27T03:09:23.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.667+0000 7f1916ffd640 4 set_mon_vals no callback set 2024-03-27T03:09:23.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.669+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f190c01ac00 con 0x7f1910006410 2024-03-27T03:09:23.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.669+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f190c0848f0 con 0x7f1910006410 2024-03-27T03:09:23.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.669+0000 7f19157fa640 10 client.4391.objecter ms_dispatch 0x7f1918000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:09:23.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.670+0000 7f19157fa640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:09:23.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.670+0000 7f19157fa640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:09:23.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.670+0000 7f19157fa640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-27T03:09:23.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.670+0000 7f1917fff640 10 monclient: get_auth_request con 0x7f18fc072340 auth_method 0 2024-03-27T03:09:23.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.670+0000 7f19157fa640 10 client.4391.objecter ms_handle_connect 0x7f18fc072340 2024-03-27T03:09:23.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.671+0000 7f191df66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:23.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.671+0000 7f191df66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:23.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.671+0000 7f191df66640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:23.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.671+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f8005080 con 0x7f1910006410 2024-03-27T03:09:23.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.681+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f190c050230 con 0x7f1910006410 2024-03-27T03:09:23.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.682+0000 7f19157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:23.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.682+0000 7f19157fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:09:23.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.995+0000 7f191df66640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:23.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.995+0000 7f191df66640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:23.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.995+0000 7f191df66640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:23.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.995+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18f8005330 con 0x7f1910006410 2024-03-27T03:09:23.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.999+0000 7f19157fa640 1 -- 172.21.15.153:0/2455811721 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f190c083fa0 con 0x7f1910006410 2024-03-27T03:09:23.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.999+0000 7f19157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:23.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:23.999+0000 7f19157fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:09:23.992 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:09:23.992 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:09:23.997 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.003+0000 7f191df66640 10 librados: watch_flush enter 2024-03-27T03:09:23.997 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.003+0000 7f191df66640 10 librados: watch_flush exit 2024-03-27T03:09:23.997 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.003+0000 7f191df66640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-27T03:09:23.997 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.003+0000 7f191df66640 10 client.4391.objecter successfully canceled tick 2024-03-27T03:09:23.997 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.003+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 >> v1:172.21.15.153:6816/2637937781 conn(0x7f18fc072340 legacy=0x7f18fc074800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:23.997 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.003+0000 7f191df66640 10 monclient: shutdown 2024-03-27T03:09:23.997 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.003+0000 7f191df66640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:23.997 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.003+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 >> v1:172.21.15.153:6790/0 conn(0x7f1910006410 legacy=0x7f191015dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:23.997 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.003+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 shutdown_connections 2024-03-27T03:09:23.997 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.003+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 >> 172.21.15.153:0/2455811721 conn(0x7f19100058f0 msgr2=0x7f19100043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:23.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.005+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 shutdown_connections 2024-03-27T03:09:23.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.006+0000 7f191df66640 1 -- 172.21.15.153:0/2455811721 wait complete. 2024-03-27T03:09:23.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:24.006+0000 7f191df66640 1 librados: shutdown 2024-03-27T03:09:24.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:09:30.826 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:09:30.985 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.992+0000 7f03da802640 10 monclient: get_monmap_and_config 2024-03-27T03:09:30.985 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.992+0000 7f03da802640 10 monclient: build_initial_monmap 2024-03-27T03:09:30.985 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.992+0000 7f03da802640 10 monclient: monmap: 2024-03-27T03:09:30.985 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:30.985 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:30.985 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:09:30.993237+0000 2024-03-27T03:09:30.985 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:09:30.993237+0000 2024-03-27T03:09:30.986 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:30.986 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:30.986 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:30.986 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:09:30.986 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:09:30.986 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:30.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.994+0000 7f03da802640 1 Processor -- start 2024-03-27T03:09:30.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.994+0000 7f03da802640 1 -- start start 2024-03-27T03:09:30.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.994+0000 7f03da802640 10 monclient: init 2024-03-27T03:09:30.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.995+0000 7f03da802640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:30.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.995+0000 7f03da802640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:09:30.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.995+0000 7f03da802640 10 monclient(hunting): picked mon.noname-b con 0x7f03d40afb50 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:30.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.996+0000 7f03da802640 10 monclient(hunting): picked mon.noname-a con 0x7f03d4141880 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:30.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.996+0000 7f03da802640 10 monclient(hunting): picked mon.noname-c con 0x7f03d406fce0 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:30.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.996+0000 7f03da802640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d4071060 con 0x7f03d4141880 2024-03-27T03:09:30.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.996+0000 7f03da802640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d4141d10 con 0x7f03d406fce0 2024-03-27T03:09:30.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.996+0000 7f03da802640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d406f3a0 con 0x7f03d40afb50 2024-03-27T03:09:30.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.996+0000 7f03da802640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:30.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.996+0000 7f03da802640 10 monclient(hunting): authenticate will time out at 1531.301916s 2024-03-27T03:09:30.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.999+0000 7f03d9800640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f03d40afb50 0x7f03d4142230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:39025/0 (socket says 172.21.15.153:39025) 2024-03-27T03:09:30.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:30.999+0000 7f03d9800640 1 -- 172.21.15.153:0/1966450178 learned_addr learned my addr 172.21.15.153:0/1966450178 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:30.996 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.002+0000 7f03cbfff640 1 -- 172.21.15.153:0/1966450178 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157060814 0 0) 0x7f03d406f3a0 con 0x7f03d40afb50 2024-03-27T03:09:30.996 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.002+0000 7f03cbfff640 10 monclient(hunting): _negotiate 2024-03-27T03:09:30.996 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.002+0000 7f03cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:30.996 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.002+0000 7f03cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:30.996 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.002+0000 7f03cbfff640 10 monclient(hunting): my global_id is 4394 2024-03-27T03:09:30.996 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.002+0000 7f03cbfff640 1 -- 172.21.15.153:0/1966450178 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03c0003440 con 0x7f03d40afb50 2024-03-27T03:09:30.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 1 -- 172.21.15.153:0/1966450178 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 263292778 0 0) 0x7f03c0003440 con 0x7f03d40afb50 2024-03-27T03:09:30.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 1 -- 172.21.15.153:0/1966450178 >> v1:172.21.15.196:6789/0 conn(0x7f03d406fce0 legacy=0x7f03d406ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 1 -- 172.21.15.153:0/1966450178 >> v1:172.21.15.153:6789/0 conn(0x7f03d4141880 legacy=0x7f03d406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 10 monclient: _finish_hunting 0 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 1 monclient: found mon.noname-b 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 1 -- 172.21.15.153:0/1966450178 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d406f530 con 0x7f03d40afb50 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 10 monclient: _finish_auth 0 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 10 monclient: _check_auth_tickets 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 1 -- 172.21.15.153:0/1966450178 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f03cc001a10 con 0x7f03d40afb50 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.004+0000 7f03cbfff640 10 monclient: dump: 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:31.000 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.005+0000 7f03cbfff640 1 -- 172.21.15.153:0/1966450178 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03cc001c30 con 0x7f03d40afb50 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.005+0000 7f03cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.005+0000 7f03cbfff640 1 -- 172.21.15.153:0/1966450178 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f03cc001fc0 con 0x7f03d40afb50 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.005+0000 7f03cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.005+0000 7f03cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.005+0000 7f03cbfff640 10 monclient: dump: 2024-03-27T03:09:31.001 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.005+0000 7f03da802640 5 monclient: authenticate success, global_id 4394 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.005+0000 7f03da802640 10 monclient: get_monmap_and_config success 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.005+0000 7f03da802640 4 set_mon_vals no callback set 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.006+0000 7f03da802640 10 monclient: shutdown 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.006+0000 7f03da802640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.006+0000 7f03da802640 1 -- 172.21.15.153:0/1966450178 >> v1:172.21.15.153:6790/0 conn(0x7f03d40afb50 legacy=0x7f03d4142230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:31.002 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.006+0000 7f03da802640 1 -- 172.21.15.153:0/1966450178 shutdown_connections 2024-03-27T03:09:31.003 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.006+0000 7f03da802640 1 -- 172.21.15.153:0/1966450178 >> 172.21.15.153:0/1966450178 conn(0x7f03d40b09d0 msgr2=0x7f03d40b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:31.003 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.007+0000 7f03da802640 1 -- 172.21.15.153:0/1966450178 shutdown_connections 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.009+0000 7f03da802640 1 -- 172.21.15.153:0/1966450178 wait complete. 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.010+0000 7f03da802640 10 monclient: build_initial_monmap 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.010+0000 7f03da802640 10 monclient: monmap: 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.011+0000 7f03da802640 1 librados: starting msgr at 2024-03-27T03:09:31.004 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.011+0000 7f03da802640 1 librados: starting objecter 2024-03-27T03:09:31.005 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.011+0000 7f03da802640 1 Processor -- start 2024-03-27T03:09:31.006 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 1 -- start start 2024-03-27T03:09:31.013 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 1 librados: setting wanted keys 2024-03-27T03:09:31.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 1 librados: calling monclient init 2024-03-27T03:09:31.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 10 monclient: init 2024-03-27T03:09:31.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:31.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:09:31.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 10 monclient(hunting): picked mon.noname-b con 0x7f03d406fce0 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:31.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 10 monclient(hunting): picked mon.noname-c con 0x7f03d40afb50 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:31.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 10 monclient(hunting): picked mon.noname-a con 0x7f03d413c180 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:31.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d413f0d0 con 0x7f03d413c180 2024-03-27T03:09:31.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d413e290 con 0x7f03d406fce0 2024-03-27T03:09:31.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d41d33a0 con 0x7f03d40afb50 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.013+0000 7f03da802640 10 monclient(hunting): authenticate will time out at 1531.319441s 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.018+0000 7f03d9800640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f03d406fce0 0x7f03d41d2c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:50673/0 (socket says 172.21.15.153:50673) 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.018+0000 7f03d8fff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f03d40afb50 0x7f03d4135c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:39131/0 (socket says 172.21.15.153:39131) 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.018+0000 7f03d8fff640 1 -- 172.21.15.153:0/1754407816 learned_addr learned my addr 172.21.15.153:0/1754407816 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 10 client.?.objecter ms_handle_connect 0x7f03d40afb50 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 10 client.?.objecter ms_handle_connect 0x7f03d406fce0 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271727027 0 0) 0x7f03d41d33a0 con 0x7f03d40afb50 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:31.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 10 monclient(hunting): my global_id is 4397 2024-03-27T03:09:31.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b8004eb0 con 0x7f03d40afb50 2024-03-27T03:09:31.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020254144 0 0) 0x7f03d413e290 con 0x7f03d406fce0 2024-03-27T03:09:31.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:09:31.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:31.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:31.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 10 monclient(hunting): my global_id is 4384 2024-03-27T03:09:31.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.019+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b8006230 con 0x7f03d406fce0 2024-03-27T03:09:31.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.020+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2843673096 0 0) 0x7f03b8004eb0 con 0x7f03d40afb50 2024-03-27T03:09:31.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.020+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03d413e290 con 0x7f03d40afb50 2024-03-27T03:09:31.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.020+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f03d0001930 con 0x7f03d40afb50 2024-03-27T03:09:31.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.020+0000 7f03c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:31.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.020+0000 7f03c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:09:31.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.020+0000 7f03c9ffb640 10 monclient(hunting): dump: 2024-03-27T03:09:31.017 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:31.017 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:31.017 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:31.017 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:31.017 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:31.017 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:31.017 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:31.017 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:31.017 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:31.017 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:31.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.020+0000 7f03c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:09:31.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3719848132 0 0) 0x7f03d413e290 con 0x7f03d40afb50 2024-03-27T03:09:31.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 >> v1:172.21.15.196:6789/0 conn(0x7f03d406fce0 legacy=0x7f03d41d2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:31.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 >> v1:172.21.15.153:6789/0 conn(0x7f03d413c180 legacy=0x7f03d4136360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:31.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:09:31.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 1 monclient: found mon.c 2024-03-27T03:09:31.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:31.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:31.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d4136a70 con 0x7f03d40afb50 2024-03-27T03:09:31.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 10 monclient: _finish_auth 0 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03d0003610 con 0x7f03d40afb50 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.023+0000 7f03c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f03d00039e0 con 0x7f03d40afb50 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03c9ffb640 10 monclient: dump: 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:31.021 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03caffd640 4 set_mon_vals no callback set 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03da802640 5 monclient: authenticate success, global_id 4397 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03da802640 10 monclient: _renew_subs 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03da802640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03da802640 1 -- 172.21.15.153:0/1754407816 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d413c640 con 0x7f03d40afb50 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03da802640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03da802640 10 monclient: _renew_subs 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03da802640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03da802640 1 -- 172.21.15.153:0/1754407816 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d41ca370 con 0x7f03d40afb50 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.024+0000 7f03da802640 1 librados: init done 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.026+0000 7f03da802640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:31.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.026+0000 7f03da802640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:31.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.026+0000 7f03da802640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:31.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.026+0000 7f03da802640 1 -- 172.21.15.153:0/1754407816 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d404f730 con 0x7f03d40afb50 2024-03-27T03:09:31.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.029+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f03d0003bc0 con 0x7f03d40afb50 2024-03-27T03:09:31.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.030+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f03d00848d0 con 0x7f03d40afb50 2024-03-27T03:09:31.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.030+0000 7f03c9ffb640 10 client.4397.objecter ms_dispatch 0x7f03d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:09:31.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.030+0000 7f03c9ffb640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:09:31.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.030+0000 7f03c9ffb640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:09:31.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.030+0000 7f03c9ffb640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-27T03:09:31.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.030+0000 7f03d9800640 10 monclient: get_auth_request con 0x7f03b806df10 auth_method 0 2024-03-27T03:09:31.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.031+0000 7f03c9ffb640 10 client.4397.objecter ms_handle_connect 0x7f03b806df10 2024-03-27T03:09:31.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.038+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03d00501b0 con 0x7f03d40afb50 2024-03-27T03:09:31.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.038+0000 7f03c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:31.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.038+0000 7f03c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:09:31.351 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.358+0000 7f03da802640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:31.352 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.358+0000 7f03da802640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:31.352 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.358+0000 7f03da802640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:09:31.352 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.358+0000 7f03da802640 1 -- 172.21.15.153:0/1754407816 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03d413c7d0 con 0x7f03d40afb50 2024-03-27T03:09:31.352 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.359+0000 7f03c9ffb640 1 -- 172.21.15.153:0/1754407816 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f03d00501b0 con 0x7f03d40afb50 2024-03-27T03:09:31.352 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.359+0000 7f03c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:31.352 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.359+0000 7f03c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:09:31.355 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:09:31.356 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:09:31.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.362+0000 7f03da802640 10 librados: watch_flush enter 2024-03-27T03:09:31.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.363+0000 7f03da802640 10 librados: watch_flush exit 2024-03-27T03:09:31.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.363+0000 7f03da802640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-27T03:09:31.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.363+0000 7f03da802640 10 client.4397.objecter successfully canceled tick 2024-03-27T03:09:31.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.364+0000 7f03da802640 1 -- 172.21.15.153:0/1754407816 >> v1:172.21.15.153:6816/2637937781 conn(0x7f03b806df10 legacy=0x7f03b80703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:31.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.364+0000 7f03da802640 10 monclient: shutdown 2024-03-27T03:09:31.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.364+0000 7f03da802640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:31.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.364+0000 7f03da802640 1 -- 172.21.15.153:0/1754407816 >> v1:172.21.15.153:6790/0 conn(0x7f03d40afb50 legacy=0x7f03d4135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:31.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.364+0000 7f03da802640 1 -- 172.21.15.153:0/1754407816 shutdown_connections 2024-03-27T03:09:31.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.364+0000 7f03da802640 1 -- 172.21.15.153:0/1754407816 >> 172.21.15.153:0/1754407816 conn(0x7f03d40b09d0 msgr2=0x7f03d406ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:31.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.365+0000 7f03da802640 1 -- 172.21.15.153:0/1754407816 shutdown_connections 2024-03-27T03:09:31.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.365+0000 7f03da802640 1 -- 172.21.15.153:0/1754407816 wait complete. 2024-03-27T03:09:31.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:31.365+0000 7f03da802640 1 librados: shutdown 2024-03-27T03:09:31.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:09:38.185 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:09:38.342 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.348+0000 7f2ce5132640 10 monclient: get_monmap_and_config 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.348+0000 7f2ce5132640 10 monclient: build_initial_monmap 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.348+0000 7f2ce5132640 10 monclient: monmap: 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:09:38.349113+0000 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:09:38.349113+0000 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:38.343 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.349+0000 7f2ce5132640 1 Processor -- start 2024-03-27T03:09:38.344 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.349+0000 7f2ce5132640 1 -- start start 2024-03-27T03:09:38.344 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.349+0000 7f2ce5132640 10 monclient: init 2024-03-27T03:09:38.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.350+0000 7f2ce5132640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:38.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.350+0000 7f2ce5132640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:09:38.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.350+0000 7f2ce5132640 10 monclient(hunting): picked mon.noname-c con 0x7f2ce006d9d0 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:38.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.350+0000 7f2ce5132640 10 monclient(hunting): picked mon.noname-a con 0x7f2ce013ef30 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:38.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.350+0000 7f2ce5132640 10 monclient(hunting): picked mon.noname-b con 0x7f2ce00b07d0 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:38.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.350+0000 7f2ce5132640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce00aed10 con 0x7f2ce013ef30 2024-03-27T03:09:38.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.350+0000 7f2ce5132640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce00b0c40 con 0x7f2ce006d9d0 2024-03-27T03:09:38.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.350+0000 7f2ce5132640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce00ab1b0 con 0x7f2ce00b07d0 2024-03-27T03:09:38.346 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.350+0000 7f2ce5132640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:38.346 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.350+0000 7f2ce5132640 10 monclient(hunting): authenticate will time out at 1538.656352s 2024-03-27T03:09:38.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.355+0000 7f2cded9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f2ce006d9d0 0x7f2ce013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:56771/0 (socket says 172.21.15.153:56771) 2024-03-27T03:09:38.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.355+0000 7f2cded9d640 1 -- 172.21.15.153:0/1008782901 learned_addr learned my addr 172.21.15.153:0/1008782901 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:38.351 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.358+0000 7f2cddd9b640 1 -- 172.21.15.153:0/1008782901 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653482443 0 0) 0x7f2ce00b0c40 con 0x7f2ce006d9d0 2024-03-27T03:09:38.351 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.358+0000 7f2cddd9b640 10 monclient(hunting): _negotiate 2024-03-27T03:09:38.351 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.358+0000 7f2cddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:38.351 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.358+0000 7f2cddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:38.351 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.358+0000 7f2cddd9b640 10 monclient(hunting): my global_id is 4387 2024-03-27T03:09:38.351 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.358+0000 7f2cddd9b640 1 -- 172.21.15.153:0/1008782901 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ccc003520 con 0x7f2ce006d9d0 2024-03-27T03:09:38.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 1 -- 172.21.15.153:0/1008782901 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528033945 0 0) 0x7f2ce00aed10 con 0x7f2ce013ef30 2024-03-27T03:09:38.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient(hunting): _negotiate 2024-03-27T03:09:38.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient(hunting): my global_id is 4374 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 1 -- 172.21.15.153:0/1008782901 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ce00b0c40 con 0x7f2ce013ef30 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 1 -- 172.21.15.153:0/1008782901 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 236941076 0 0) 0x7f2ce00b0c40 con 0x7f2ce013ef30 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 1 -- 172.21.15.153:0/1008782901 >> v1:172.21.15.153:6790/0 conn(0x7f2ce00b07d0 legacy=0x7f2ce014bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 1 -- 172.21.15.153:0/1008782901 >> v1:172.21.15.196:6789/0 conn(0x7f2ce006d9d0 legacy=0x7f2ce013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient: _finish_hunting 0 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 1 monclient: found mon.noname-a 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 1 -- 172.21.15.153:0/1008782901 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce00ab490 con 0x7f2ce013ef30 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient: _finish_auth 0 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient: _check_auth_tickets 2024-03-27T03:09:38.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 1 -- 172.21.15.153:0/1008782901 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2cd0001670 con 0x7f2ce013ef30 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient: dump: 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2ce5132640 5 monclient: authenticate success, global_id 4374 2024-03-27T03:09:38.357 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2ce5132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 1 -- 172.21.15.153:0/1008782901 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cd0001850 con 0x7f2ce013ef30 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2ce5132640 10 monclient: get_monmap_and_config success 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2ce5132640 4 set_mon_vals no callback set 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 1 -- 172.21.15.153:0/1008782901 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2cd0001c60 con 0x7f2ce013ef30 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.359+0000 7f2cddd9b640 10 monclient: dump: 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:38.358 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:38.359 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:38.359 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:38.359 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:38.359 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:38.359 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.360+0000 7f2ce5132640 10 monclient: shutdown 2024-03-27T03:09:38.359 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.360+0000 7f2ce5132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:38.359 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.360+0000 7f2ce5132640 1 -- 172.21.15.153:0/1008782901 >> v1:172.21.15.153:6789/0 conn(0x7f2ce013ef30 legacy=0x7f2ce00af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:38.359 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.360+0000 7f2ce5132640 1 -- 172.21.15.153:0/1008782901 shutdown_connections 2024-03-27T03:09:38.359 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.360+0000 7f2ce5132640 1 -- 172.21.15.153:0/1008782901 >> 172.21.15.153:0/1008782901 conn(0x7f2ce006b660 msgr2=0x7f2ce006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:38.359 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.361+0000 7f2ce5132640 1 -- 172.21.15.153:0/1008782901 shutdown_connections 2024-03-27T03:09:38.359 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.362+0000 7f2ce5132640 1 -- 172.21.15.153:0/1008782901 wait complete. 2024-03-27T03:09:38.360 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.363+0000 7f2ce5132640 10 monclient: build_initial_monmap 2024-03-27T03:09:38.360 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.363+0000 7f2ce5132640 10 monclient: monmap: 2024-03-27T03:09:38.360 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:38.360 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:09:38.360 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:09:38.361 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:09:38.361 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:38.361 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:38.361 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:38.361 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:09:38.361 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:09:38.361 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:38.361 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.364+0000 7f2ce5132640 1 librados: starting msgr at 2024-03-27T03:09:38.361 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.364+0000 7f2ce5132640 1 librados: starting objecter 2024-03-27T03:09:38.361 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.364+0000 7f2ce5132640 1 Processor -- start 2024-03-27T03:09:38.361 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.366+0000 7f2ce5132640 1 -- start start 2024-03-27T03:09:38.363 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.366+0000 7f2ce5132640 1 librados: setting wanted keys 2024-03-27T03:09:38.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.366+0000 7f2ce5132640 1 librados: calling monclient init 2024-03-27T03:09:38.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.366+0000 7f2ce5132640 10 monclient: init 2024-03-27T03:09:38.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.367+0000 7f2ce5132640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:38.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.367+0000 7f2ce5132640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:09:38.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.367+0000 7f2ce5132640 10 monclient(hunting): picked mon.noname-a con 0x7f2ce00b07d0 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:38.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.367+0000 7f2ce5132640 10 monclient(hunting): picked mon.noname-c con 0x7f2ce013ef30 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:38.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.367+0000 7f2ce5132640 10 monclient(hunting): picked mon.noname-b con 0x7f2ce013aa80 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:38.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.367+0000 7f2ce5132640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce0135c10 con 0x7f2ce00b07d0 2024-03-27T03:09:38.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.367+0000 7f2ce5132640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce0137010 con 0x7f2ce013aa80 2024-03-27T03:09:38.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.367+0000 7f2ce5132640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce0135580 con 0x7f2ce013ef30 2024-03-27T03:09:38.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.367+0000 7f2ce5132640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:38.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.367+0000 7f2ce5132640 10 monclient(hunting): authenticate will time out at 1538.673000s 2024-03-27T03:09:38.370 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.374+0000 7f2cdf59e640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f2ce013aa80 0x7f2ce01df4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:56849/0 (socket says 172.21.15.153:56849) 2024-03-27T03:09:38.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.374+0000 7f2cdf59e640 1 -- 172.21.15.153:0/4266473258 learned_addr learned my addr 172.21.15.153:0/4266473258 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:38.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.374+0000 7f2cd77fe640 10 client.?.objecter ms_handle_connect 0x7f2ce013aa80 2024-03-27T03:09:38.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.374+0000 7f2cd77fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 10 client.?.objecter ms_handle_connect 0x7f2ce013ef30 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185258231 0 0) 0x7f2ce0137010 con 0x7f2ce013aa80 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 10 monclient(hunting): _negotiate 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 10 monclient(hunting): my global_id is 4390 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc0004eb0 con 0x7f2ce013aa80 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138138739 0 0) 0x7f2ce0135580 con 0x7f2ce013ef30 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 10 monclient(hunting): _negotiate 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:38.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 10 monclient(hunting): my global_id is 4400 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.380+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ce0137010 con 0x7f2ce013ef30 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.381+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3427702916 0 0) 0x7f2cc0004eb0 con 0x7f2ce013aa80 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.381+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ce0135580 con 0x7f2ce013aa80 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.381+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2cd8001950 con 0x7f2ce013aa80 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.381+0000 7f2cd77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.381+0000 7f2cd77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.381+0000 7f2cd77fe640 10 monclient(hunting): dump: 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:38.377 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:38.378 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:38.378 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:38.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.381+0000 7f2cd77fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 10 client.?.objecter ms_handle_connect 0x7f2ce00b07d0 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4065937542 0 0) 0x7f2ce0137010 con 0x7f2ce013ef30 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cc0004eb0 con 0x7f2ce013ef30 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1407218133 0 0) 0x7f2ce0135580 con 0x7f2ce013aa80 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 >> v1:172.21.15.153:6790/0 conn(0x7f2ce013ef30 legacy=0x7f2ce01ded90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 >> v1:172.21.15.153:6789/0 conn(0x7f2ce00b07d0 legacy=0x7f2ce014dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 10 monclient: _finish_hunting 0 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 1 monclient: found mon.b 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce0135710 con 0x7f2ce013aa80 2024-03-27T03:09:38.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 10 monclient: _finish_auth 0 2024-03-27T03:09:38.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 10 monclient: _check_auth_tickets 2024-03-27T03:09:38.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:38.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2ce5132640 5 monclient: authenticate success, global_id 4390 2024-03-27T03:09:38.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2ce5132640 10 monclient: _renew_subs 2024-03-27T03:09:38.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2ce5132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:09:38.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2ce5132640 1 -- 172.21.15.153:0/4266473258 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce0135900 con 0x7f2ce013aa80 2024-03-27T03:09:38.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2ce5132640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2ce5132640 10 monclient: _renew_subs 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2ce5132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2ce5132640 1 -- 172.21.15.153:0/4266473258 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce0134d80 con 0x7f2ce013aa80 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2ce5132640 1 librados: init done 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cd8003610 con 0x7f2ce013aa80 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2cd80039a0 con 0x7f2ce013aa80 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.386+0000 7f2cdcd99640 4 set_mon_vals no callback set 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.387+0000 7f2cd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.387+0000 7f2cd77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.387+0000 7f2cd77fe640 10 monclient: dump: 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:38.381 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:38.382 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:38.382 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:38.382 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:38.382 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:38.382 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:38.382 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:38.382 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:38.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.390+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f2cd8003b00 con 0x7f2ce013aa80 2024-03-27T03:09:38.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.390+0000 7f2ce5132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:38.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.390+0000 7f2ce5132640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:38.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.390+0000 7f2ce5132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:09:38.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.390+0000 7f2ce5132640 1 -- 172.21.15.153:0/4266473258 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc4005080 con 0x7f2ce013aa80 2024-03-27T03:09:38.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.390+0000 7f2cded9d640 10 monclient: get_auth_request con 0x7f2cc006dd40 auth_method 0 2024-03-27T03:09:38.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.391+0000 7f2cd77fe640 10 client.4390.objecter ms_handle_connect 0x7f2cc006dd40 2024-03-27T03:09:38.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.391+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f2cd80849e0 con 0x7f2ce013aa80 2024-03-27T03:09:38.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.391+0000 7f2cd77fe640 10 client.4390.objecter ms_dispatch 0x7f2ce0000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:09:38.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.391+0000 7f2cd77fe640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:09:38.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.391+0000 7f2cd77fe640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:09:38.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.391+0000 7f2cd77fe640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-27T03:09:38.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.399+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2cd80502c0 con 0x7f2ce013aa80 2024-03-27T03:09:38.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.399+0000 7f2cd77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:38.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.399+0000 7f2cd77fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:09:38.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.728+0000 7f2ce5132640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:38.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.728+0000 7f2ce5132640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:38.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.728+0000 7f2ce5132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:09:38.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.728+0000 7f2ce5132640 1 -- 172.21.15.153:0/4266473258 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cc4005330 con 0x7f2ce013aa80 2024-03-27T03:09:38.722 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.728+0000 7f2cd77fe640 1 -- 172.21.15.153:0/4266473258 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f2cd80521f0 con 0x7f2ce013aa80 2024-03-27T03:09:38.722 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.728+0000 7f2cd77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:38.722 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.729+0000 7f2cd77fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:09:38.722 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:09:38.722 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:09:38.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.732+0000 7f2ce5132640 10 librados: watch_flush enter 2024-03-27T03:09:38.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.732+0000 7f2ce5132640 10 librados: watch_flush exit 2024-03-27T03:09:38.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.732+0000 7f2ce5132640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-27T03:09:38.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.732+0000 7f2ce5132640 10 client.4390.objecter successfully canceled tick 2024-03-27T03:09:38.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.732+0000 7f2ce5132640 1 -- 172.21.15.153:0/4266473258 >> v1:172.21.15.153:6816/2637937781 conn(0x7f2cc006dd40 legacy=0x7f2cc0070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:38.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.732+0000 7f2ce5132640 10 monclient: shutdown 2024-03-27T03:09:38.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.732+0000 7f2ce5132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:38.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.733+0000 7f2ce5132640 1 -- 172.21.15.153:0/4266473258 >> v1:172.21.15.196:6789/0 conn(0x7f2ce013aa80 legacy=0x7f2ce01df4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:38.729 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.736+0000 7f2ce5132640 1 -- 172.21.15.153:0/4266473258 shutdown_connections 2024-03-27T03:09:38.729 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.737+0000 7f2ce5132640 1 -- 172.21.15.153:0/4266473258 >> 172.21.15.153:0/4266473258 conn(0x7f2ce006b660 msgr2=0x7f2ce00abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:38.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.738+0000 7f2ce5132640 1 -- 172.21.15.153:0/4266473258 shutdown_connections 2024-03-27T03:09:38.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.738+0000 7f2ce5132640 1 -- 172.21.15.153:0/4266473258 wait complete. 2024-03-27T03:09:38.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:38.738+0000 7f2ce5132640 1 librados: shutdown 2024-03-27T03:09:38.749 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:09:45.555 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:09:45.704 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.710+0000 7f9d5f120640 10 monclient: get_monmap_and_config 2024-03-27T03:09:45.704 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.710+0000 7f9d5f120640 10 monclient: build_initial_monmap 2024-03-27T03:09:45.704 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.710+0000 7f9d5f120640 10 monclient: monmap: 2024-03-27T03:09:45.705 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:45.705 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:45.705 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:09:45.712237+0000 2024-03-27T03:09:45.705 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:09:45.712237+0000 2024-03-27T03:09:45.705 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:45.705 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:45.705 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:45.705 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:09:45.705 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:09:45.705 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:45.708 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.713+0000 7f9d5f120640 1 Processor -- start 2024-03-27T03:09:45.708 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.714+0000 7f9d5f120640 1 -- start start 2024-03-27T03:09:45.708 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.714+0000 7f9d5f120640 10 monclient: init 2024-03-27T03:09:45.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.715+0000 7f9d5f120640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:45.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.715+0000 7f9d5f120640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:09:45.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.715+0000 7f9d5f120640 10 monclient(hunting): picked mon.noname-a con 0x7f9d5002c070 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:45.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.715+0000 7f9d5f120640 10 monclient(hunting): picked mon.noname-c con 0x7f9d500c81a0 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:45.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.715+0000 7f9d5f120640 10 monclient(hunting): picked mon.noname-b con 0x7f9d500c1a30 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:45.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.715+0000 7f9d5f120640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500c1ea0 con 0x7f9d5002c070 2024-03-27T03:09:45.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.715+0000 7f9d5f120640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5002cf40 con 0x7f9d500c81a0 2024-03-27T03:09:45.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.715+0000 7f9d5f120640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5002d220 con 0x7f9d500c1a30 2024-03-27T03:09:45.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.715+0000 7f9d5f120640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:45.716 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.715+0000 7f9d5f120640 10 monclient(hunting): authenticate will time out at 1546.022032s 2024-03-27T03:09:45.716 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.720+0000 7f9d5d91d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f9d500c81a0 0x7f9d500c1320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:49201/0 (socket says 172.21.15.153:49201) 2024-03-27T03:09:45.716 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.720+0000 7f9d5d91d640 1 -- 172.21.15.153:0/3517533460 learned_addr learned my addr 172.21.15.153:0/3517533460 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:45.718 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.721+0000 7f9d5d11c640 1 -- 172.21.15.153:0/3517533460 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188650094 0 0) 0x7f9d5002cf40 con 0x7f9d500c81a0 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.721+0000 7f9d5d11c640 10 monclient(hunting): _negotiate 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.721+0000 7f9d5d11c640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.721+0000 7f9d5d11c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.721+0000 7f9d5d11c640 10 monclient(hunting): my global_id is 4393 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.721+0000 7f9d5d11c640 1 -- 172.21.15.153:0/3517533460 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d44003520 con 0x7f9d500c81a0 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 1 -- 172.21.15.153:0/3517533460 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1515800641 0 0) 0x7f9d44003520 con 0x7f9d500c81a0 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 1 -- 172.21.15.153:0/3517533460 >> v1:172.21.15.153:6790/0 conn(0x7f9d500c1a30 legacy=0x7f9d500d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 1 -- 172.21.15.153:0/3517533460 >> v1:172.21.15.153:6789/0 conn(0x7f9d5002c070 legacy=0x7f9d500c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 10 monclient: _finish_hunting 0 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 1 monclient: found mon.noname-c 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 1 -- 172.21.15.153:0/3517533460 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d500cae10 con 0x7f9d500c81a0 2024-03-27T03:09:45.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 10 monclient: _finish_auth 0 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 10 monclient: _check_auth_tickets 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 1 -- 172.21.15.153:0/3517533460 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9d48001690 con 0x7f9d500c81a0 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5d11c640 10 monclient: dump: 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:45.720 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5f120640 5 monclient: authenticate success, global_id 4393 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.722+0000 7f9d5f120640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.723+0000 7f9d5d11c640 1 -- 172.21.15.153:0/3517533460 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d480017f0 con 0x7f9d500c81a0 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.723+0000 7f9d5d11c640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.723+0000 7f9d5d11c640 1 -- 172.21.15.153:0/3517533460 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9d48001c00 con 0x7f9d500c81a0 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.723+0000 7f9d5d11c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.723+0000 7f9d5d11c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.723+0000 7f9d5d11c640 10 monclient: dump: 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:45.721 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:45.722 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:45.722 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:45.722 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:45.722 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.723+0000 7f9d5f120640 10 monclient: get_monmap_and_config success 2024-03-27T03:09:45.722 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.723+0000 7f9d5f120640 4 set_mon_vals no callback set 2024-03-27T03:09:45.723 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.724+0000 7f9d5f120640 10 monclient: shutdown 2024-03-27T03:09:45.723 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.724+0000 7f9d5f120640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.724+0000 7f9d5f120640 1 -- 172.21.15.153:0/3517533460 >> v1:172.21.15.196:6789/0 conn(0x7f9d500c81a0 legacy=0x7f9d500c1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.726+0000 7f9d5f120640 1 -- 172.21.15.153:0/3517533460 shutdown_connections 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.726+0000 7f9d5f120640 1 -- 172.21.15.153:0/3517533460 >> 172.21.15.153:0/3517533460 conn(0x7f9d50006490 msgr2=0x7f9d500068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.727+0000 7f9d5f120640 1 -- 172.21.15.153:0/3517533460 shutdown_connections 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.728+0000 7f9d5f120640 1 -- 172.21.15.153:0/3517533460 wait complete. 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.729+0000 7f9d5f120640 10 monclient: build_initial_monmap 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.729+0000 7f9d5f120640 10 monclient: monmap: 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:45.724 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:45.725 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:09:45.725 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:09:45.725 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:45.725 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.729+0000 7f9d5f120640 1 librados: starting msgr at 2024-03-27T03:09:45.725 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.730+0000 7f9d5f120640 1 librados: starting objecter 2024-03-27T03:09:45.725 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.730+0000 7f9d5f120640 1 Processor -- start 2024-03-27T03:09:45.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.730+0000 7f9d5f120640 1 -- start start 2024-03-27T03:09:45.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.730+0000 7f9d5f120640 1 librados: setting wanted keys 2024-03-27T03:09:45.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.730+0000 7f9d5f120640 1 librados: calling monclient init 2024-03-27T03:09:45.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.730+0000 7f9d5f120640 10 monclient: init 2024-03-27T03:09:45.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.731+0000 7f9d5f120640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:45.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.731+0000 7f9d5f120640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:09:45.728 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.734+0000 7f9d5f120640 10 monclient(hunting): picked mon.noname-a con 0x7f9d5002c070 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:45.728 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.734+0000 7f9d5f120640 10 monclient(hunting): picked mon.noname-b con 0x7f9d500c1a30 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:45.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.737+0000 7f9d5d91d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f9d500c1a30 0x7f9d500c4bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:49303/0 (socket says 172.21.15.153:49303) 2024-03-27T03:09:45.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.737+0000 7f9d5f120640 10 monclient(hunting): picked mon.noname-c con 0x7f9d500c81a0 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:45.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.737+0000 7f9d5f120640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500b5460 con 0x7f9d5002c070 2024-03-27T03:09:45.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.737+0000 7f9d5f120640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5002ebe0 con 0x7f9d500c1a30 2024-03-27T03:09:45.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.737+0000 7f9d5f120640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5002ee50 con 0x7f9d500c81a0 2024-03-27T03:09:45.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.737+0000 7f9d5f120640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:45.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.737+0000 7f9d5f120640 10 monclient(hunting): authenticate will time out at 1546.044461s 2024-03-27T03:09:45.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.737+0000 7f9d5e11e640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f9d5002c070 0x7f9d5002d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:50463/0 (socket says 172.21.15.153:50463) 2024-03-27T03:09:45.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.737+0000 7f9d5e11e640 1 -- 172.21.15.153:0/4289728483 learned_addr learned my addr 172.21.15.153:0/4289728483 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:45.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.738+0000 7f9d4effd640 10 client.?.objecter ms_handle_connect 0x7f9d5002c070 2024-03-27T03:09:45.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.738+0000 7f9d4effd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:45.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.738+0000 7f9d4effd640 10 client.?.objecter ms_handle_connect 0x7f9d500c1a30 2024-03-27T03:09:45.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.738+0000 7f9d4effd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:45.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.739+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866232905 0 0) 0x7f9d500b5460 con 0x7f9d5002c070 2024-03-27T03:09:45.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.740+0000 7f9d4effd640 10 monclient(hunting): _negotiate 2024-03-27T03:09:45.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.740+0000 7f9d4effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:45.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.740+0000 7f9d4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:45.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.740+0000 7f9d4effd640 10 monclient(hunting): my global_id is 4383 2024-03-27T03:09:45.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.740+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d3c004eb0 con 0x7f9d5002c070 2024-03-27T03:09:45.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.740+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217478855 0 0) 0x7f9d5002ebe0 con 0x7f9d500c1a30 2024-03-27T03:09:45.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.740+0000 7f9d4effd640 10 monclient(hunting): _negotiate 2024-03-27T03:09:45.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.740+0000 7f9d4effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:45.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.740+0000 7f9d4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:45.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.740+0000 7f9d4effd640 10 monclient(hunting): my global_id is 4396 2024-03-27T03:09:45.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.740+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d500b5460 con 0x7f9d500c1a30 2024-03-27T03:09:45.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.741+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 25698238 0 0) 0x7f9d500b5460 con 0x7f9d500c1a30 2024-03-27T03:09:45.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.741+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d5002ebe0 con 0x7f9d500c1a30 2024-03-27T03:09:45.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.741+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9d48001b00 con 0x7f9d500c1a30 2024-03-27T03:09:45.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.741+0000 7f9d4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:45.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.741+0000 7f9d4effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:09:45.747 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.741+0000 7f9d4effd640 10 monclient(hunting): dump: 2024-03-27T03:09:45.747 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:45.747 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:45.747 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:45.747 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:45.747 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:45.748 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:45.748 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:45.748 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:45.748 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:45.748 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:45.748 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.741+0000 7f9d4effd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:09:45.748 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3961075966 0 0) 0x7f9d5002ebe0 con 0x7f9d500c1a30 2024-03-27T03:09:45.749 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 >> v1:172.21.15.153:6790/0 conn(0x7f9d500c81a0 legacy=0x7f9d50167900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:45.749 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 >> v1:172.21.15.153:6789/0 conn(0x7f9d5002c070 legacy=0x7f9d5002d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:45.749 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d4effd640 10 monclient: _finish_hunting 0 2024-03-27T03:09:45.749 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d4effd640 1 monclient: found mon.b 2024-03-27T03:09:45.749 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:45.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d4effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:09:45.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d500c7080 con 0x7f9d500c1a30 2024-03-27T03:09:45.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d4effd640 10 monclient: _finish_auth 0 2024-03-27T03:09:45.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d4effd640 10 monclient: _check_auth_tickets 2024-03-27T03:09:45.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:45.750 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d5f120640 5 monclient: authenticate success, global_id 4396 2024-03-27T03:09:45.751 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d5f120640 10 monclient: _renew_subs 2024-03-27T03:09:45.751 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d5f120640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:09:45.751 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d5f120640 1 -- 172.21.15.153:0/4289728483 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d500b5f10 con 0x7f9d500c1a30 2024-03-27T03:09:45.751 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d5f120640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:09:45.751 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d5f120640 10 monclient: _renew_subs 2024-03-27T03:09:45.751 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d5f120640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:09:45.752 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d5f120640 1 -- 172.21.15.153:0/4289728483 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d500c67c0 con 0x7f9d500c1a30 2024-03-27T03:09:45.752 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.742+0000 7f9d5f120640 1 librados: init done 2024-03-27T03:09:45.752 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.743+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d48001d40 con 0x7f9d500c1a30 2024-03-27T03:09:45.752 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.743+0000 7f9d4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:45.752 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.743+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9d48004c30 con 0x7f9d500c1a30 2024-03-27T03:09:45.752 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.743+0000 7f9d4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:45.752 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.743+0000 7f9d4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:09:45.752 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.743+0000 7f9d4effd640 10 monclient: dump: 2024-03-27T03:09:45.752 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:45.753 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:45.753 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:45.753 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:45.753 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:45.753 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:45.753 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:45.753 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:45.753 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:45.754 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:45.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.744+0000 7f9d5c91b640 4 set_mon_vals no callback set 2024-03-27T03:09:45.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.746+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f9d4801ac00 con 0x7f9d500c1a30 2024-03-27T03:09:45.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.747+0000 7f9d5f120640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:45.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.747+0000 7f9d5f120640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:45.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.747+0000 7f9d5f120640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:09:45.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.747+0000 7f9d5f120640 1 -- 172.21.15.153:0/4289728483 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d40005080 con 0x7f9d500c1a30 2024-03-27T03:09:45.754 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.747+0000 7f9d5e11e640 10 monclient: get_auth_request con 0x7f9d3c06dcc0 auth_method 0 2024-03-27T03:09:45.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.748+0000 7f9d4effd640 10 client.4396.objecter ms_handle_connect 0x7f9d3c06dcc0 2024-03-27T03:09:45.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.748+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f9d48084ad0 con 0x7f9d500c1a30 2024-03-27T03:09:45.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.748+0000 7f9d4effd640 10 client.4396.objecter ms_dispatch 0x7f9d58000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:09:45.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.748+0000 7f9d4effd640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:09:45.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.748+0000 7f9d4effd640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:09:45.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.748+0000 7f9d4effd640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-27T03:09:45.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.754+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d480503b0 con 0x7f9d500c1a30 2024-03-27T03:09:45.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.754+0000 7f9d4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:45.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:45.754+0000 7f9d4effd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:09:46.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.101+0000 7f9d5f120640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:46.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.101+0000 7f9d5f120640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:46.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.101+0000 7f9d5f120640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:09:46.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.101+0000 7f9d5f120640 1 -- 172.21.15.153:0/4289728483 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d40005330 con 0x7f9d500c1a30 2024-03-27T03:09:46.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.102+0000 7f9d4effd640 1 -- 172.21.15.153:0/4289728483 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f9d480522e0 con 0x7f9d500c1a30 2024-03-27T03:09:46.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.102+0000 7f9d4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:46.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.102+0000 7f9d4effd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:09:46.098 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:09:46.098 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:09:46.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.106+0000 7f9d5f120640 10 librados: watch_flush enter 2024-03-27T03:09:46.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.106+0000 7f9d5f120640 10 librados: watch_flush exit 2024-03-27T03:09:46.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.106+0000 7f9d5f120640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-27T03:09:46.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.106+0000 7f9d5f120640 10 client.4396.objecter successfully canceled tick 2024-03-27T03:09:46.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.106+0000 7f9d5f120640 1 -- 172.21.15.153:0/4289728483 >> v1:172.21.15.153:6816/2637937781 conn(0x7f9d3c06dcc0 legacy=0x7f9d3c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:46.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.106+0000 7f9d5f120640 10 monclient: shutdown 2024-03-27T03:09:46.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.106+0000 7f9d5f120640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:46.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.106+0000 7f9d5f120640 1 -- 172.21.15.153:0/4289728483 >> v1:172.21.15.196:6789/0 conn(0x7f9d500c1a30 legacy=0x7f9d500c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:46.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.106+0000 7f9d5f120640 1 -- 172.21.15.153:0/4289728483 shutdown_connections 2024-03-27T03:09:46.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.106+0000 7f9d5f120640 1 -- 172.21.15.153:0/4289728483 >> 172.21.15.153:0/4289728483 conn(0x7f9d50006490 msgr2=0x7f9d50029d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:46.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.106+0000 7f9d5f120640 1 -- 172.21.15.153:0/4289728483 shutdown_connections 2024-03-27T03:09:46.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.107+0000 7f9d5f120640 1 -- 172.21.15.153:0/4289728483 wait complete. 2024-03-27T03:09:46.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:46.107+0000 7f9d5f120640 1 librados: shutdown 2024-03-27T03:09:46.115 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:09:52.921 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:09:53.074 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.081+0000 7f8cb17c9640 10 monclient: get_monmap_and_config 2024-03-27T03:09:53.078 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.081+0000 7f8cb17c9640 10 monclient: build_initial_monmap 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.081+0000 7f8cb17c9640 10 monclient: monmap: 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:09:53.082174+0000 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:09:53.082174+0000 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.082+0000 7f8cb17c9640 1 Processor -- start 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.082+0000 7f8cb17c9640 1 -- start start 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.082+0000 7f8cb17c9640 10 monclient: init 2024-03-27T03:09:53.079 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.083+0000 7f8cb17c9640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.083+0000 7f8cb17c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.083+0000 7f8cb17c9640 10 monclient(hunting): picked mon.noname-c con 0x7f8cac0a9930 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.083+0000 7f8cb17c9640 10 monclient(hunting): picked mon.noname-a con 0x7f8cac149dc0 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.083+0000 7f8caa7fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8cac149dc0 0x7f8cac14a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:38310/0 (socket says 172.21.15.153:38310) 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.083+0000 7f8cb17c9640 10 monclient(hunting): picked mon.noname-b con 0x7f8cac14ff40 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8cb17c9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac0aae60 con 0x7f8cac149dc0 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8cb17c9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac0b4310 con 0x7f8cac0a9930 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8cb17c9640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac0afdd0 con 0x7f8cac14ff40 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8cb17c9640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8cb17c9640 10 monclient(hunting): authenticate will time out at 1553.389641s 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8caa7fc640 1 -- 172.21.15.153:0/489780297 learned_addr learned my addr 172.21.15.153:0/489780297 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8ca9ffb640 1 -- 172.21.15.153:0/489780297 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550530079 0 0) 0x7f8cac0aae60 con 0x7f8cac149dc0 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8ca9ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8ca9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8ca9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:53.080 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8ca9ffb640 10 monclient(hunting): my global_id is 4386 2024-03-27T03:09:53.081 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8ca9ffb640 1 -- 172.21.15.153:0/489780297 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c94003520 con 0x7f8cac149dc0 2024-03-27T03:09:53.081 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8ca9ffb640 1 -- 172.21.15.153:0/489780297 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1081539930 0 0) 0x7f8c94003520 con 0x7f8cac149dc0 2024-03-27T03:09:53.081 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.084+0000 7f8ca9ffb640 1 -- 172.21.15.153:0/489780297 >> v1:172.21.15.153:6790/0 conn(0x7f8cac14ff40 legacy=0x7f8cac152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:09:53.085 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.086+0000 7f8ca9ffb640 1 -- 172.21.15.153:0/489780297 >> v1:172.21.15.196:6789/0 conn(0x7f8cac0a9930 legacy=0x7f8cac143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:09:53.085 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8ca9ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:09:53.085 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8ca9ffb640 1 monclient: found mon.noname-a 2024-03-27T03:09:53.085 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8ca9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:53.085 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8ca9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:09:53.085 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8ca9ffb640 1 -- 172.21.15.153:0/489780297 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac14a950 con 0x7f8cac149dc0 2024-03-27T03:09:53.085 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8ca9ffb640 10 monclient: _finish_auth 0 2024-03-27T03:09:53.085 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8ca9ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8ca9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8ca9ffb640 1 -- 172.21.15.153:0/489780297 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8c9c001690 con 0x7f8cac149dc0 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8ca9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8ca9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8ca9ffb640 10 monclient: dump: 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:53.086 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8cb17c9640 5 monclient: authenticate success, global_id 4386 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.087+0000 7f8cb17c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.088+0000 7f8ca9ffb640 1 -- 172.21.15.153:0/489780297 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c9c001860 con 0x7f8cac149dc0 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.088+0000 7f8ca9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.088+0000 7f8ca9ffb640 1 -- 172.21.15.153:0/489780297 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8c9c001cb0 con 0x7f8cac149dc0 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.088+0000 7f8ca9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.088+0000 7f8ca9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.088+0000 7f8ca9ffb640 10 monclient: dump: 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:53.087 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.088+0000 7f8cb17c9640 10 monclient: get_monmap_and_config success 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.088+0000 7f8cb17c9640 4 set_mon_vals no callback set 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.089+0000 7f8cb17c9640 10 monclient: shutdown 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.089+0000 7f8cb17c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.089+0000 7f8cb17c9640 1 -- 172.21.15.153:0/489780297 >> v1:172.21.15.153:6789/0 conn(0x7f8cac149dc0 legacy=0x7f8cac14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.089+0000 7f8cb17c9640 1 -- 172.21.15.153:0/489780297 shutdown_connections 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.089+0000 7f8cb17c9640 1 -- 172.21.15.153:0/489780297 >> 172.21.15.153:0/489780297 conn(0x7f8cac0b35e0 msgr2=0x7f8cac0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.089+0000 7f8cb17c9640 1 -- 172.21.15.153:0/489780297 shutdown_connections 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.089+0000 7f8cb17c9640 1 -- 172.21.15.153:0/489780297 wait complete. 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.090+0000 7f8cb17c9640 10 monclient: build_initial_monmap 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.091+0000 7f8cb17c9640 10 monclient: monmap: 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:09:53.088 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.091+0000 7f8cb17c9640 1 librados: starting msgr at 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.091+0000 7f8cb17c9640 1 librados: starting objecter 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.092+0000 7f8cb17c9640 1 Processor -- start 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.092+0000 7f8cb17c9640 1 -- start start 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.092+0000 7f8cb17c9640 1 librados: setting wanted keys 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.092+0000 7f8cb17c9640 1 librados: calling monclient init 2024-03-27T03:09:53.089 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.092+0000 7f8cb17c9640 10 monclient: init 2024-03-27T03:09:53.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.093+0000 7f8cb17c9640 10 monclient: _reopen_session rank -1 2024-03-27T03:09:53.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.093+0000 7f8cb17c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:09:53.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.093+0000 7f8cb17c9640 10 monclient(hunting): picked mon.noname-a con 0x7f8cac0a9930 addr v1:172.21.15.153:6789/0 2024-03-27T03:09:53.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.093+0000 7f8cb17c9640 10 monclient(hunting): picked mon.noname-b con 0x7f8cac149dc0 addr v1:172.21.15.196:6789/0 2024-03-27T03:09:53.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.096+0000 7f8caaffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8cac0a9930 0x7f8cac1de9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:42740/0 (socket says 172.21.15.153:42740) 2024-03-27T03:09:53.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.096+0000 7f8cb17c9640 10 monclient(hunting): picked mon.noname-c con 0x7f8cac14ff40 addr v1:172.21.15.153:6790/0 2024-03-27T03:09:53.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.096+0000 7f8cb17c9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac13bd30 con 0x7f8cac0a9930 2024-03-27T03:09:53.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.096+0000 7f8cb17c9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac145c00 con 0x7f8cac149dc0 2024-03-27T03:09:53.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.096+0000 7f8cb17c9640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac149100 con 0x7f8cac14ff40 2024-03-27T03:09:53.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.096+0000 7f8cb17c9640 10 monclient(hunting): _renew_subs 2024-03-27T03:09:53.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.096+0000 7f8cb17c9640 10 monclient(hunting): authenticate will time out at 1553.402289s 2024-03-27T03:09:53.093 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.096+0000 7f8caaffd640 1 -- 172.21.15.153:0/1516267946 learned_addr learned my addr 172.21.15.153:0/1516267946 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:09:53.093 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.097+0000 7f8c937fe640 10 client.?.objecter ms_handle_connect 0x7f8cac0a9930 2024-03-27T03:09:53.093 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.097+0000 7f8c937fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:53.093 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.097+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014438018 0 0) 0x7f8cac13bd30 con 0x7f8cac0a9930 2024-03-27T03:09:53.093 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.097+0000 7f8c937fe640 10 monclient(hunting): _negotiate 2024-03-27T03:09:53.093 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.097+0000 7f8c937fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:53.093 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.097+0000 7f8c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:53.093 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.097+0000 7f8c937fe640 10 monclient(hunting): my global_id is 4389 2024-03-27T03:09:53.094 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.100+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c98004eb0 con 0x7f8cac0a9930 2024-03-27T03:09:53.094 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.100+0000 7f8c937fe640 10 client.?.objecter ms_handle_connect 0x7f8cac149dc0 2024-03-27T03:09:53.094 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.100+0000 7f8c937fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:53.094 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.101+0000 7f8c937fe640 10 client.?.objecter ms_handle_connect 0x7f8cac14ff40 2024-03-27T03:09:53.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.101+0000 7f8c937fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:09:53.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.101+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634151344 0 0) 0x7f8cac149100 con 0x7f8cac14ff40 2024-03-27T03:09:53.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.101+0000 7f8c937fe640 10 monclient(hunting): _negotiate 2024-03-27T03:09:53.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.101+0000 7f8c937fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:53.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.101+0000 7f8c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:53.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.101+0000 7f8c937fe640 10 monclient(hunting): my global_id is 4403 2024-03-27T03:09:53.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.101+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cac13bd30 con 0x7f8cac14ff40 2024-03-27T03:09:53.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225207892 0 0) 0x7f8cac145c00 con 0x7f8cac149dc0 2024-03-27T03:09:53.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 10 monclient(hunting): _negotiate 2024-03-27T03:09:53.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:09:53.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:09:53.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 10 monclient(hunting): my global_id is 4399 2024-03-27T03:09:53.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cac149100 con 0x7f8cac149dc0 2024-03-27T03:09:53.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1614694876 0 0) 0x7f8c98004eb0 con 0x7f8cac0a9930 2024-03-27T03:09:53.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cac145c00 con 0x7f8cac0a9930 2024-03-27T03:09:53.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8ca0001870 con 0x7f8cac0a9930 2024-03-27T03:09:53.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:53.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:09:53.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 10 monclient(hunting): dump: 2024-03-27T03:09:53.096 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:53.096 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:53.097 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:53.097 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:53.097 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:53.097 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:53.097 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:53.097 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:53.097 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:53.097 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:53.097 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.102+0000 7f8c937fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:09:53.097 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3244688039 0 0) 0x7f8cac149100 con 0x7f8cac149dc0 2024-03-27T03:09:53.097 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c98004eb0 con 0x7f8cac149dc0 2024-03-27T03:09:53.098 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8c9c001b70 con 0x7f8cac149dc0 2024-03-27T03:09:53.098 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:53.098 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-27T03:09:53.098 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 10 monclient(hunting): dump: 2024-03-27T03:09:53.098 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:53.098 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:53.098 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:53.098 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:53.098 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:53.098 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:53.098 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:53.098 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:53.099 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:53.099 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:53.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2471078983 0 0) 0x7f8cac145c00 con 0x7f8cac0a9930 2024-03-27T03:09:53.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 >> v1:172.21.15.153:6790/0 conn(0x7f8cac14ff40 legacy=0x7f8cac13b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:53.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 >> v1:172.21.15.196:6789/0 conn(0x7f8cac149dc0 legacy=0x7f8cac13af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:53.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 10 monclient: _finish_hunting 0 2024-03-27T03:09:53.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 1 monclient: found mon.a 2024-03-27T03:09:53.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:09:53.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:53.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac149290 con 0x7f8cac0a9930 2024-03-27T03:09:53.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 10 monclient: _finish_auth 0 2024-03-27T03:09:53.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 10 monclient: _check_auth_tickets 2024-03-27T03:09:53.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8c937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:09:53.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8cb17c9640 5 monclient: authenticate success, global_id 4389 2024-03-27T03:09:53.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8cb17c9640 10 monclient: _renew_subs 2024-03-27T03:09:53.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8cb17c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:53.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8cb17c9640 1 -- 172.21.15.153:0/1516267946 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac144b60 con 0x7f8cac0a9930 2024-03-27T03:09:53.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8cb17c9640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:09:53.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8cb17c9640 10 monclient: _renew_subs 2024-03-27T03:09:53.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8cb17c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:53.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8cb17c9640 1 -- 172.21.15.153:0/1516267946 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac0ac4c0 con 0x7f8cac0a9930 2024-03-27T03:09:53.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.103+0000 7f8cb17c9640 1 librados: init done 2024-03-27T03:09:53.102 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.105+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ca0001d80 con 0x7f8cac0a9930 2024-03-27T03:09:53.102 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.105+0000 7f8c937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:09:53.102 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.105+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8ca0004aa0 con 0x7f8cac0a9930 2024-03-27T03:09:53.102 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.105+0000 7f8c937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:09:53.102 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.105+0000 7f8c937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:09:53.102 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.105+0000 7f8ca97fa640 4 set_mon_vals no callback set 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.105+0000 7f8c937fe640 10 monclient: dump: 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.105+0000 7f8c917fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.106+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f8ca0004c40 con 0x7f8cac0a9930 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.106+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f8ca0084a10 con 0x7f8cac0a9930 2024-03-27T03:09:53.103 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.106+0000 7f8c937fe640 10 client.4389.objecter ms_dispatch 0x7f8cac000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:09:53.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.106+0000 7f8c937fe640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:09:53.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.106+0000 7f8c937fe640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:09:53.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.107+0000 7f8c937fe640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-27T03:09:53.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.107+0000 7f8c917fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:53.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.107+0000 7f8c917fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:53.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.107+0000 7f8c917fa640 1 -- 172.21.15.153:0/1516267946 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cac1499b0 con 0x7f8cac0a9930 2024-03-27T03:09:53.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.107+0000 7f8caa7fc640 10 monclient: get_auth_request con 0x7f8c9806dfb0 auth_method 0 2024-03-27T03:09:53.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.107+0000 7f8c937fe640 10 client.4389.objecter ms_handle_connect 0x7f8c9806dfb0 2024-03-27T03:09:53.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.115+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ca00502f0 con 0x7f8cac0a9930 2024-03-27T03:09:53.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.115+0000 7f8c937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:09:53.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.115+0000 7f8c937fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:09:53.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.454+0000 7f8c917fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:53.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.454+0000 7f8c917fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:53.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.454+0000 7f8c917fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:09:53.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.454+0000 7f8c917fa640 1 -- 172.21.15.153:0/1516267946 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cac144eb0 con 0x7f8cac0a9930 2024-03-27T03:09:53.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.455+0000 7f8c937fe640 1 -- 172.21.15.153:0/1516267946 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f8ca0052220 con 0x7f8cac0a9930 2024-03-27T03:09:53.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.455+0000 7f8c937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:09:53.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.455+0000 7f8c937fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:09:53.448 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:09:53.448 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:09:53.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.459+0000 7f8c917fa640 10 librados: watch_flush enter 2024-03-27T03:09:53.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.459+0000 7f8c917fa640 10 librados: watch_flush exit 2024-03-27T03:09:53.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.459+0000 7f8c917fa640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-27T03:09:53.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.459+0000 7f8c917fa640 10 client.4389.objecter successfully canceled tick 2024-03-27T03:09:53.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.459+0000 7f8c917fa640 1 -- 172.21.15.153:0/1516267946 >> v1:172.21.15.153:6816/2637937781 conn(0x7f8c9806dfb0 legacy=0x7f8c98070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:53.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.459+0000 7f8c917fa640 10 monclient: shutdown 2024-03-27T03:09:53.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.459+0000 7f8c917fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:09:53.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.459+0000 7f8c917fa640 1 -- 172.21.15.153:0/1516267946 >> v1:172.21.15.153:6789/0 conn(0x7f8cac0a9930 legacy=0x7f8cac1de9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:09:53.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.460+0000 7f8c917fa640 1 -- 172.21.15.153:0/1516267946 shutdown_connections 2024-03-27T03:09:53.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.460+0000 7f8c917fa640 1 -- 172.21.15.153:0/1516267946 >> 172.21.15.153:0/1516267946 conn(0x7f8cac0b35e0 msgr2=0x7f8cac0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:09:53.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.461+0000 7f8c917fa640 1 -- 172.21.15.153:0/1516267946 shutdown_connections 2024-03-27T03:09:53.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.461+0000 7f8c917fa640 1 -- 172.21.15.153:0/1516267946 wait complete. 2024-03-27T03:09:53.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:09:53.461+0000 7f8c917fa640 1 librados: shutdown 2024-03-27T03:09:53.476 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:10:00.282 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:10:00.435 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.441+0000 7f4ac8033640 10 monclient: get_monmap_and_config 2024-03-27T03:10:00.435 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.441+0000 7f4ac8033640 10 monclient: build_initial_monmap 2024-03-27T03:10:00.435 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.441+0000 7f4ac8033640 10 monclient: monmap: 2024-03-27T03:10:00.435 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:00.435 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:00.435 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:10:00.442853+0000 2024-03-27T03:10:00.435 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:10:00.442853+0000 2024-03-27T03:10:00.435 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:00.435 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:00.435 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:00.435 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:10:00.436 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:10:00.436 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:00.436 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.443+0000 7f4ac8033640 1 Processor -- start 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.443+0000 7f4ac8033640 1 -- start start 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.443+0000 7f4ac8033640 10 monclient: init 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.444+0000 7f4ac8033640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.444+0000 7f4ac8033640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.444+0000 7f4ac8033640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac00b0770 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.444+0000 7f4ac8033640 10 monclient(hunting): picked mon.noname-c con 0x7f4ac014a700 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.444+0000 7f4ac8033640 10 monclient(hunting): picked mon.noname-a con 0x7f4ac00ac5a0 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.444+0000 7f4ac8033640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac00aba70 con 0x7f4ac00ac5a0 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.444+0000 7f4ac5dcf640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f4ac00b0770 0x7f4ac0149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:59080/0 (socket says 172.21.15.153:59080) 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.444+0000 7f4ac5dcf640 1 -- 172.21.15.153:0/1543456988 learned_addr learned my addr 172.21.15.153:0/1543456988 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.444+0000 7f4ac8033640 1 -- 172.21.15.153:0/1543456988 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac00aecb0 con 0x7f4ac014a700 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.445+0000 7f4ac8033640 1 -- 172.21.15.153:0/1543456988 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac00af150 con 0x7f4ac00b0770 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.445+0000 7f4ac8033640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.445+0000 7f4ac8033640 10 monclient(hunting): authenticate will time out at 1560.750937s 2024-03-27T03:10:00.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.445+0000 7f4ac4dcd640 1 -- 172.21.15.153:0/1543456988 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392313715 0 0) 0x7f4ac00aecb0 con 0x7f4ac014a700 2024-03-27T03:10:00.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.445+0000 7f4ac4dcd640 10 monclient(hunting): _negotiate 2024-03-27T03:10:00.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.445+0000 7f4ac4dcd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:00.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.445+0000 7f4ac4dcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:00.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.445+0000 7f4ac4dcd640 10 monclient(hunting): my global_id is 4402 2024-03-27T03:10:00.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.445+0000 7f4ac4dcd640 1 -- 172.21.15.153:0/1543456988 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aac003520 con 0x7f4ac014a700 2024-03-27T03:10:00.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.445+0000 7f4ac4dcd640 1 -- 172.21.15.153:0/1543456988 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029861095 0 0) 0x7f4ac00af150 con 0x7f4ac00b0770 2024-03-27T03:10:00.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient(hunting): _negotiate 2024-03-27T03:10:00.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:00.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:00.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient(hunting): my global_id is 4406 2024-03-27T03:10:00.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 1 -- 172.21.15.153:0/1543456988 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac00aecb0 con 0x7f4ac00b0770 2024-03-27T03:10:00.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 1 -- 172.21.15.153:0/1543456988 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3494077863 0 0) 0x7f4aac003520 con 0x7f4ac014a700 2024-03-27T03:10:00.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 1 -- 172.21.15.153:0/1543456988 >> v1:172.21.15.153:6790/0 conn(0x7f4ac00b0770 legacy=0x7f4ac0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:00.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 1 -- 172.21.15.153:0/1543456988 >> v1:172.21.15.153:6789/0 conn(0x7f4ac00ac5a0 legacy=0x7f4ac0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:10:00.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient: _finish_hunting 0 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 1 monclient: found mon.noname-c 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 1 -- 172.21.15.153:0/1543456988 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac00ac9d0 con 0x7f4ac014a700 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient: _finish_auth 0 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient: _check_auth_tickets 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac8033640 5 monclient: authenticate success, global_id 4402 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac8033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 1 -- 172.21.15.153:0/1543456988 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4ab00016b0 con 0x7f4ac014a700 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient: dump: 2024-03-27T03:10:00.442 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac8033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 1 -- 172.21.15.153:0/1543456988 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ab0001900 con 0x7f4ac014a700 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac8033640 10 monclient: get_monmap_and_config success 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac8033640 4 set_mon_vals no callback set 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 1 -- 172.21.15.153:0/1543456988 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4ab0001c90 con 0x7f4ac014a700 2024-03-27T03:10:00.443 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.446+0000 7f4ac4dcd640 10 monclient: dump: 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.447+0000 7f4ac8033640 10 monclient: shutdown 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.447+0000 7f4ac8033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:00.444 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.447+0000 7f4ac8033640 1 -- 172.21.15.153:0/1543456988 >> v1:172.21.15.196:6789/0 conn(0x7f4ac014a700 legacy=0x7f4ac00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:00.445 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.447+0000 7f4ac8033640 1 -- 172.21.15.153:0/1543456988 shutdown_connections 2024-03-27T03:10:00.445 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.447+0000 7f4ac8033640 1 -- 172.21.15.153:0/1543456988 >> 172.21.15.153:0/1543456988 conn(0x7f4ac006b660 msgr2=0x7f4ac006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:00.445 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.447+0000 7f4ac8033640 1 -- 172.21.15.153:0/1543456988 shutdown_connections 2024-03-27T03:10:00.445 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.447+0000 7f4ac8033640 1 -- 172.21.15.153:0/1543456988 wait complete. 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.449+0000 7f4ac8033640 10 monclient: build_initial_monmap 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.449+0000 7f4ac8033640 10 monclient: monmap: 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:00.449 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.449+0000 7f4ac8033640 1 librados: starting msgr at 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.449+0000 7f4ac8033640 1 librados: starting objecter 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.449+0000 7f4ac8033640 1 Processor -- start 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.450+0000 7f4ac8033640 1 -- start start 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.450+0000 7f4ac8033640 1 librados: setting wanted keys 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.450+0000 7f4ac8033640 1 librados: calling monclient init 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.450+0000 7f4ac8033640 10 monclient: init 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.451+0000 7f4ac8033640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.451+0000 7f4ac8033640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.451+0000 7f4ac8033640 10 monclient(hunting): picked mon.noname-a con 0x7f4ac00ac5a0 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.451+0000 7f4ac8033640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac00b0770 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.451+0000 7f4ac8033640 10 monclient(hunting): picked mon.noname-c con 0x7f4ac0126f00 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.451+0000 7f4ac8033640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac01de9d0 con 0x7f4ac00ac5a0 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.451+0000 7f4ac8033640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac0145b90 con 0x7f4ac00b0770 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.451+0000 7f4ac8033640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac01239d0 con 0x7f4ac0126f00 2024-03-27T03:10:00.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.451+0000 7f4ac8033640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.451+0000 7f4ac8033640 10 monclient(hunting): authenticate will time out at 1560.757044s 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.451+0000 7f4ac55ce640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f4ac00b0770 0x7f4ac0144fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:43602/0 (socket says 172.21.15.153:43602) 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.451+0000 7f4ac55ce640 1 -- 172.21.15.153:0/2482195422 learned_addr learned my addr 172.21.15.153:0/2482195422 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 client.?.objecter ms_handle_connect 0x7f4ac00b0770 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 client.?.objecter ms_handle_connect 0x7f4ac00ac5a0 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87537461 0 0) 0x7f4ac0145b90 con 0x7f4ac00b0770 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 monclient(hunting): _negotiate 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 monclient(hunting): my global_id is 4405 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa4004eb0 con 0x7f4ac00b0770 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494416798 0 0) 0x7f4ac01de9d0 con 0x7f4ac00ac5a0 2024-03-27T03:10:00.451 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 monclient(hunting): _negotiate 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 monclient(hunting): my global_id is 4392 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac0145b90 con 0x7f4ac00ac5a0 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.452+0000 7f4ab67fc640 10 client.?.objecter ms_handle_connect 0x7f4ac0126f00 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2983089703 0 0) 0x7f4aa4004eb0 con 0x7f4ac00b0770 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ac01de9d0 con 0x7f4ac00b0770 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126621342 0 0) 0x7f4ac01239d0 con 0x7f4ac0126f00 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 10 monclient(hunting): _negotiate 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 10 monclient(hunting): my global_id is 4409 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa4004eb0 con 0x7f4ac0126f00 2024-03-27T03:10:00.452 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4ab0001c10 con 0x7f4ac00b0770 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 10 monclient(hunting): dump: 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.453+0000 7f4ab67fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1664910400 0 0) 0x7f4ac01de9d0 con 0x7f4ac00b0770 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 >> v1:172.21.15.153:6790/0 conn(0x7f4ac0126f00 legacy=0x7f4ac01dee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:00.453 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 >> v1:172.21.15.153:6789/0 conn(0x7f4ac00ac5a0 legacy=0x7f4ac01267f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ab67fc640 10 monclient: _finish_hunting 0 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ab67fc640 1 monclient: found mon.b 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ab67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ab67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac0125d10 con 0x7f4ac00b0770 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ab67fc640 10 monclient: _finish_auth 0 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ab67fc640 10 monclient: _check_auth_tickets 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ab67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ac8033640 5 monclient: authenticate success, global_id 4405 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ac8033640 10 monclient: _renew_subs 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ac8033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ac8033640 1 -- 172.21.15.153:0/2482195422 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac01456b0 con 0x7f4ac00b0770 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ac8033640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.454+0000 7f4ac8033640 10 monclient: _renew_subs 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.455+0000 7f4ac8033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:10:00.454 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.455+0000 7f4ac8033640 1 -- 172.21.15.153:0/2482195422 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac01df750 con 0x7f4ac00b0770 2024-03-27T03:10:00.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.455+0000 7f4ac8033640 1 librados: init done 2024-03-27T03:10:00.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.455+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ab0003960 con 0x7f4ac00b0770 2024-03-27T03:10:00.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.455+0000 7f4ab67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:00.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.455+0000 7f4ab77fe640 4 set_mon_vals no callback set 2024-03-27T03:10:00.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.455+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4ab0003d30 con 0x7f4ac00b0770 2024-03-27T03:10:00.456 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.456+0000 7f4ab67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:00.456 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.456+0000 7f4ab67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:10:00.456 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.456+0000 7f4ab67fc640 10 monclient: dump: 2024-03-27T03:10:00.456 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:00.456 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:00.456 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:00.456 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:00.456 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:00.456 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:00.456 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.456+0000 7f4ac8033640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.457+0000 7f4ac8033640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.457+0000 7f4ac8033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.457+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f4ab0003e90 con 0x7f4ac00b0770 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.457+0000 7f4ac8033640 1 -- 172.21.15.153:0/2482195422 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa8005080 con 0x7f4ac00b0770 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.458+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f4ab0084aa0 con 0x7f4ac00b0770 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.458+0000 7f4ab67fc640 10 client.4405.objecter ms_dispatch 0x7f4ac0000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.458+0000 7f4ab67fc640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.458+0000 7f4ab67fc640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.458+0000 7f4ab67fc640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.458+0000 7f4ac5dcf640 10 monclient: get_auth_request con 0x7f4aa406df20 auth_method 0 2024-03-27T03:10:00.457 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.459+0000 7f4ab67fc640 10 client.4405.objecter ms_handle_connect 0x7f4aa406df20 2024-03-27T03:10:00.459 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.466+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ab0050380 con 0x7f4ac00b0770 2024-03-27T03:10:00.459 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.466+0000 7f4ab67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:00.459 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.466+0000 7f4ab67fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:10:00.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.835+0000 7f4ac8033640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:00.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.835+0000 7f4ac8033640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:00.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.835+0000 7f4ac8033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:10:00.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.835+0000 7f4ac8033640 1 -- 172.21.15.153:0/2482195422 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4aa8005330 con 0x7f4ac00b0770 2024-03-27T03:10:00.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.837+0000 7f4ab67fc640 1 -- 172.21.15.153:0/2482195422 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f4ab00522b0 con 0x7f4ac00b0770 2024-03-27T03:10:00.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.837+0000 7f4ab67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:00.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.837+0000 7f4ab67fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:10:00.831 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:10:00.831 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:10:00.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.840+0000 7f4ac8033640 10 librados: watch_flush enter 2024-03-27T03:10:00.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.841+0000 7f4ac8033640 10 librados: watch_flush exit 2024-03-27T03:10:00.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.841+0000 7f4ac8033640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-27T03:10:00.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.841+0000 7f4ac8033640 10 client.4405.objecter successfully canceled tick 2024-03-27T03:10:00.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.841+0000 7f4ac8033640 1 -- 172.21.15.153:0/2482195422 >> v1:172.21.15.153:6816/2637937781 conn(0x7f4aa406df20 legacy=0x7f4aa40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:00.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.841+0000 7f4ac8033640 10 monclient: shutdown 2024-03-27T03:10:00.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.841+0000 7f4ac8033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:00.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.841+0000 7f4ac8033640 1 -- 172.21.15.153:0/2482195422 >> v1:172.21.15.196:6789/0 conn(0x7f4ac00b0770 legacy=0x7f4ac0144fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:00.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.842+0000 7f4ac8033640 1 -- 172.21.15.153:0/2482195422 shutdown_connections 2024-03-27T03:10:00.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.842+0000 7f4ac8033640 1 -- 172.21.15.153:0/2482195422 >> 172.21.15.153:0/2482195422 conn(0x7f4ac006b660 msgr2=0x7f4ac00b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:00.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.842+0000 7f4ac8033640 1 -- 172.21.15.153:0/2482195422 shutdown_connections 2024-03-27T03:10:00.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.842+0000 7f4ac8033640 1 -- 172.21.15.153:0/2482195422 wait complete. 2024-03-27T03:10:00.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:00.842+0000 7f4ac8033640 1 librados: shutdown 2024-03-27T03:10:00.851 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:10:07.658 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:10:07.797 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.803+0000 7faa7424a640 10 monclient: get_monmap_and_config 2024-03-27T03:10:07.797 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.803+0000 7faa7424a640 10 monclient: build_initial_monmap 2024-03-27T03:10:07.797 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.803+0000 7faa7424a640 10 monclient: monmap: 2024-03-27T03:10:07.797 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:07.797 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:07.797 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:10:07.804003+0000 2024-03-27T03:10:07.797 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:10:07.804003+0000 2024-03-27T03:10:07.798 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:07.798 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:07.798 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:07.798 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:10:07.798 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:10:07.798 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:07.798 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.803+0000 7faa7424a640 1 Processor -- start 2024-03-27T03:10:07.798 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.804+0000 7faa7424a640 1 -- start start 2024-03-27T03:10:07.798 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.804+0000 7faa7424a640 10 monclient: init 2024-03-27T03:10:07.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa7424a640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:07.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa7424a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:10:07.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa7424a640 10 monclient(hunting): picked mon.noname-c con 0x7faa6c0ae250 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:07.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa7424a640 10 monclient(hunting): picked mon.noname-a con 0x7faa6c13f970 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:07.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa7424a640 10 monclient(hunting): picked mon.noname-b con 0x7faa6c142800 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:07.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa7424a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6c0afe00 con 0x7faa6c13f970 2024-03-27T03:10:07.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa7424a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6c142c10 con 0x7faa6c0ae250 2024-03-27T03:10:07.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa7424a640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6c0afb20 con 0x7faa6c142800 2024-03-27T03:10:07.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa7424a640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:07.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa7424a640 10 monclient(hunting): authenticate will time out at 1568.110836s 2024-03-27T03:10:07.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa727e7640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7faa6c142800 0x7faa6c14da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:33924/0 (socket says 172.21.15.153:33924) 2024-03-27T03:10:07.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa727e7640 1 -- 172.21.15.153:0/1346196422 learned_addr learned my addr 172.21.15.153:0/1346196422 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:07.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa70fe4640 1 -- 172.21.15.153:0/1346196422 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669036519 0 0) 0x7faa6c0afb20 con 0x7faa6c142800 2024-03-27T03:10:07.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa70fe4640 10 monclient(hunting): _negotiate 2024-03-27T03:10:07.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa70fe4640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:07.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa70fe4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:07.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa70fe4640 10 monclient(hunting): my global_id is 4412 2024-03-27T03:10:07.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.805+0000 7faa70fe4640 1 -- 172.21.15.153:0/1346196422 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa58003520 con 0x7faa6c142800 2024-03-27T03:10:07.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.806+0000 7faa70fe4640 1 -- 172.21.15.153:0/1346196422 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2609707905 0 0) 0x7faa58003520 con 0x7faa6c142800 2024-03-27T03:10:07.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.806+0000 7faa70fe4640 1 -- 172.21.15.153:0/1346196422 >> v1:172.21.15.196:6789/0 conn(0x7faa6c0ae250 legacy=0x7faa6c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:07.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.806+0000 7faa70fe4640 1 -- 172.21.15.153:0/1346196422 >> v1:172.21.15.153:6789/0 conn(0x7faa6c13f970 legacy=0x7faa6c1420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:10:07.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa70fe4640 10 monclient: _finish_hunting 0 2024-03-27T03:10:07.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa70fe4640 1 monclient: found mon.noname-b 2024-03-27T03:10:07.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa70fe4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:07.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa70fe4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:10:07.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa70fe4640 1 -- 172.21.15.153:0/1346196422 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6c0aab80 con 0x7faa6c142800 2024-03-27T03:10:07.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa70fe4640 10 monclient: _finish_auth 0 2024-03-27T03:10:07.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa70fe4640 10 monclient: _check_auth_tickets 2024-03-27T03:10:07.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa70fe4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:07.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa70fe4640 1 -- 172.21.15.153:0/1346196422 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7faa640019f0 con 0x7faa6c142800 2024-03-27T03:10:07.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa70fe4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa70fe4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa70fe4640 10 monclient: dump: 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa7424a640 5 monclient: authenticate success, global_id 4412 2024-03-27T03:10:07.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.808+0000 7faa7424a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.809+0000 7faa70fe4640 1 -- 172.21.15.153:0/1346196422 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa64001c80 con 0x7faa6c142800 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.809+0000 7faa70fe4640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.809+0000 7faa70fe4640 1 -- 172.21.15.153:0/1346196422 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7faa64002050 con 0x7faa6c142800 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.809+0000 7faa70fe4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.809+0000 7faa70fe4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.809+0000 7faa70fe4640 10 monclient: dump: 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:07.806 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:07.807 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:07.807 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:07.807 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:07.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.809+0000 7faa7424a640 10 monclient: get_monmap_and_config success 2024-03-27T03:10:07.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.809+0000 7faa7424a640 4 set_mon_vals no callback set 2024-03-27T03:10:07.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.810+0000 7faa7424a640 10 monclient: shutdown 2024-03-27T03:10:07.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.810+0000 7faa7424a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:07.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.810+0000 7faa7424a640 1 -- 172.21.15.153:0/1346196422 >> v1:172.21.15.153:6790/0 conn(0x7faa6c142800 legacy=0x7faa6c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:07.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.811+0000 7faa7424a640 1 -- 172.21.15.153:0/1346196422 shutdown_connections 2024-03-27T03:10:07.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.811+0000 7faa7424a640 1 -- 172.21.15.153:0/1346196422 >> 172.21.15.153:0/1346196422 conn(0x7faa6c0b35e0 msgr2=0x7faa6c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:07.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.812+0000 7faa7424a640 1 -- 172.21.15.153:0/1346196422 shutdown_connections 2024-03-27T03:10:07.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.812+0000 7faa7424a640 1 -- 172.21.15.153:0/1346196422 wait complete. 2024-03-27T03:10:07.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.813+0000 7faa7424a640 10 monclient: build_initial_monmap 2024-03-27T03:10:07.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.813+0000 7faa7424a640 10 monclient: monmap: 2024-03-27T03:10:07.810 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.813+0000 7faa7424a640 1 librados: starting msgr at 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.813+0000 7faa7424a640 1 librados: starting objecter 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.813+0000 7faa7424a640 1 Processor -- start 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.813+0000 7faa7424a640 1 -- start start 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.813+0000 7faa7424a640 1 librados: setting wanted keys 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.813+0000 7faa7424a640 1 librados: calling monclient init 2024-03-27T03:10:07.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.813+0000 7faa7424a640 10 monclient: init 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.814+0000 7faa7424a640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.814+0000 7faa7424a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.814+0000 7faa7424a640 10 monclient(hunting): picked mon.noname-b con 0x7faa6c0ae250 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.814+0000 7faa7424a640 10 monclient(hunting): picked mon.noname-a con 0x7faa6c13f970 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.814+0000 7faa7424a640 10 monclient(hunting): picked mon.noname-c con 0x7faa6c142800 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.814+0000 7faa7424a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6c145fa0 con 0x7faa6c13f970 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.814+0000 7faa7424a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6c13a080 con 0x7faa6c0ae250 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.814+0000 7faa7424a640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6c145aa0 con 0x7faa6c142800 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.814+0000 7faa7424a640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.814+0000 7faa7424a640 10 monclient(hunting): authenticate will time out at 1568.120244s 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa71fe6640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7faa6c0ae250 0x7faa6c143470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:54488/0 (socket says 172.21.15.153:54488) 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa71fe6640 1 -- 172.21.15.153:0/821762574 learned_addr learned my addr 172.21.15.153:0/821762574 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 10 client.?.objecter ms_handle_connect 0x7faa6c0ae250 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:07.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 10 client.?.objecter ms_handle_connect 0x7faa6c142800 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589870213 0 0) 0x7faa6c145aa0 con 0x7faa6c142800 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 10 monclient(hunting): my global_id is 4415 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa50004eb0 con 0x7faa6c142800 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357348393 0 0) 0x7faa6c13a080 con 0x7faa6c0ae250 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 10 monclient(hunting): my global_id is 4411 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.815+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa6c145aa0 con 0x7faa6c0ae250 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.816+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1463550520 0 0) 0x7faa50004eb0 con 0x7faa6c142800 2024-03-27T03:10:07.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.816+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa6c13a080 con 0x7faa6c142800 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.816+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7faa64002050 con 0x7faa6c142800 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.816+0000 7faa5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.816+0000 7faa5e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.816+0000 7faa5e7fc640 10 monclient(hunting): dump: 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:07.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.816+0000 7faa5e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:10:07.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.816+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3514999425 0 0) 0x7faa6c13a080 con 0x7faa6c142800 2024-03-27T03:10:07.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.816+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 >> v1:172.21.15.196:6789/0 conn(0x7faa6c0ae250 legacy=0x7faa6c143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:07.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.816+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 >> v1:172.21.15.153:6789/0 conn(0x7faa6c13f970 legacy=0x7faa6c1db5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:10:07.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.817+0000 7faa5e7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:10:07.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.817+0000 7faa5e7fc640 1 monclient: found mon.c 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.817+0000 7faa5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.817+0000 7faa5e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.817+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6c136510 con 0x7faa6c142800 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.817+0000 7faa5e7fc640 10 monclient: _finish_auth 0 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.817+0000 7faa5e7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.817+0000 7faa5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.817+0000 7faa7424a640 5 monclient: authenticate success, global_id 4415 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.818+0000 7faa7424a640 10 monclient: _renew_subs 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.818+0000 7faa7424a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.818+0000 7faa7424a640 1 -- 172.21.15.153:0/821762574 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6c1412b0 con 0x7faa6c142800 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.818+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa64004e10 con 0x7faa6c142800 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.818+0000 7faa5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.818+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7faa64005120 con 0x7faa6c142800 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.818+0000 7faa5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:07.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.818+0000 7faa5e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.818+0000 7faa5e7fc640 10 monclient: dump: 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.818+0000 7faa5ffff640 4 set_mon_vals no callback set 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.819+0000 7faa7424a640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.819+0000 7faa7424a640 10 monclient: _renew_subs 2024-03-27T03:10:07.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.819+0000 7faa7424a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.819+0000 7faa7424a640 1 -- 172.21.15.153:0/821762574 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6c145480 con 0x7faa6c142800 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.819+0000 7faa7424a640 1 librados: init done 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.820+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7faa6401b0f0 con 0x7faa6c142800 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.820+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7faa64054dd0 con 0x7faa6c142800 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.820+0000 7faa5e7fc640 10 client.4415.objecter ms_dispatch 0x7faa6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.820+0000 7faa5e7fc640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.820+0000 7faa5e7fc640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.821+0000 7faa71fe6640 10 monclient: get_auth_request con 0x7faa5006dc10 auth_method 0 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.821+0000 7faa4ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.821+0000 7faa4ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.821+0000 7faa4ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.821+0000 7faa4ffff640 1 -- 172.21.15.153:0/821762574 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa48005080 con 0x7faa6c142800 2024-03-27T03:10:07.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.821+0000 7faa5e7fc640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-27T03:10:07.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.821+0000 7faa5e7fc640 10 client.4415.objecter ms_handle_connect 0x7faa5006dc10 2024-03-27T03:10:07.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.829+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa64050710 con 0x7faa6c142800 2024-03-27T03:10:07.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.829+0000 7faa5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:07.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:07.829+0000 7faa5e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:10:08.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.138+0000 7faa4ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:08.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.138+0000 7faa4ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:08.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.138+0000 7faa4ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:08.131 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.138+0000 7faa4ffff640 1 -- 172.21.15.153:0/821762574 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa48005330 con 0x7faa6c142800 2024-03-27T03:10:08.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.139+0000 7faa5e7fc640 1 -- 172.21.15.153:0/821762574 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7faa64052810 con 0x7faa6c142800 2024-03-27T03:10:08.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.139+0000 7faa5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:08.132 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.139+0000 7faa5e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:10:08.132 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:10:08.132 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.142+0000 7faa4ffff640 10 librados: watch_flush enter 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.142+0000 7faa4ffff640 10 librados: watch_flush exit 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.142+0000 7faa4ffff640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.142+0000 7faa4ffff640 10 client.4415.objecter successfully canceled tick 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.142+0000 7faa4ffff640 1 -- 172.21.15.153:0/821762574 >> v1:172.21.15.153:6816/2637937781 conn(0x7faa5006dc10 legacy=0x7faa500700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.142+0000 7faa4ffff640 10 monclient: shutdown 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.142+0000 7faa4ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.142+0000 7faa4ffff640 1 -- 172.21.15.153:0/821762574 >> v1:172.21.15.153:6790/0 conn(0x7faa6c142800 legacy=0x7faa6c1dbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.142+0000 7faa4ffff640 1 -- 172.21.15.153:0/821762574 shutdown_connections 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.142+0000 7faa4ffff640 1 -- 172.21.15.153:0/821762574 >> 172.21.15.153:0/821762574 conn(0x7faa6c0b35e0 msgr2=0x7faa6c0aa190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.143+0000 7faa4ffff640 1 -- 172.21.15.153:0/821762574 shutdown_connections 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.143+0000 7faa4ffff640 1 -- 172.21.15.153:0/821762574 wait complete. 2024-03-27T03:10:08.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:08.143+0000 7faa4ffff640 1 librados: shutdown 2024-03-27T03:10:08.149 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:10:14.956 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:10:15.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.099+0000 7f4ede9e9640 10 monclient: get_monmap_and_config 2024-03-27T03:10:15.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.099+0000 7f4ede9e9640 10 monclient: build_initial_monmap 2024-03-27T03:10:15.092 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.099+0000 7f4ede9e9640 10 monclient: monmap: 2024-03-27T03:10:15.092 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:15.092 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:15.093 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:10:15.100002+0000 2024-03-27T03:10:15.093 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:10:15.100002+0000 2024-03-27T03:10:15.093 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:15.093 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:15.093 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:15.093 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:10:15.093 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:10:15.093 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.100+0000 7f4ede9e9640 1 Processor -- start 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.100+0000 7f4ede9e9640 1 -- start start 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.100+0000 7f4ede9e9640 10 monclient: init 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ede9e9640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ede9e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ede9e9640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed80a9750 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ede9e9640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed813e1e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ede9e9640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed8071c30 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ede9e9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed80ab300 con 0x7f4ed813e1e0 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ede9e9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed80ac2f0 con 0x7f4ed8071c30 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ede9e9640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed80ad8d0 con 0x7f4ed80a9750 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ede9e9640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ede9e9640 10 monclient(hunting): authenticate will time out at 1575.406799s 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ed7fff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f4ed80a9750 0x7f4ed813dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:35374/0 (socket says 172.21.15.153:35374) 2024-03-27T03:10:15.095 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ed7fff640 1 -- 172.21.15.153:0/3347342446 learned_addr learned my addr 172.21.15.153:0/3347342446 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:15.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ed77fe640 1 -- 172.21.15.153:0/3347342446 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279169258 0 0) 0x7f4ed80ad8d0 con 0x7f4ed80a9750 2024-03-27T03:10:15.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ed77fe640 10 monclient(hunting): _negotiate 2024-03-27T03:10:15.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ed77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:15.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:15.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ed77fe640 10 monclient(hunting): my global_id is 4418 2024-03-27T03:10:15.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.101+0000 7f4ed77fe640 1 -- 172.21.15.153:0/3347342446 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec4003520 con 0x7f4ed80a9750 2024-03-27T03:10:15.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.102+0000 7f4ed77fe640 1 -- 172.21.15.153:0/3347342446 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2489923489 0 0) 0x7f4ec4003520 con 0x7f4ed80a9750 2024-03-27T03:10:15.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.102+0000 7f4ed77fe640 1 -- 172.21.15.153:0/3347342446 >> v1:172.21.15.196:6789/0 conn(0x7f4ed8071c30 legacy=0x7f4ed80ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:15.096 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.102+0000 7f4ed77fe640 1 -- 172.21.15.153:0/3347342446 >> v1:172.21.15.153:6789/0 conn(0x7f4ed813e1e0 legacy=0x7f4ed80ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ed77fe640 10 monclient: _finish_hunting 0 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ed77fe640 1 monclient: found mon.noname-b 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ed77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ed77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ed77fe640 1 -- 172.21.15.153:0/3347342446 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed80ada60 con 0x7f4ed80a9750 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ed77fe640 10 monclient: _finish_auth 0 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ed77fe640 10 monclient: _check_auth_tickets 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ed77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ed77fe640 1 -- 172.21.15.153:0/3347342446 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4ec8001690 con 0x7f4ed80a9750 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ed77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ed77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ed77fe640 10 monclient: dump: 2024-03-27T03:10:15.099 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ede9e9640 5 monclient: authenticate success, global_id 4418 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.104+0000 7f4ede9e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ed77fe640 1 -- 172.21.15.153:0/3347342446 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ec8001860 con 0x7f4ed80a9750 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ed77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:15.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ede9e9640 10 monclient: get_monmap_and_config success 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ede9e9640 4 set_mon_vals no callback set 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ed77fe640 1 -- 172.21.15.153:0/3347342446 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4ec8001cb0 con 0x7f4ed80a9750 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ed77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ed77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ed77fe640 10 monclient: dump: 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ede9e9640 10 monclient: shutdown 2024-03-27T03:10:15.101 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ede9e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:15.102 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ede9e9640 1 -- 172.21.15.153:0/3347342446 >> v1:172.21.15.153:6790/0 conn(0x7f4ed80a9750 legacy=0x7f4ed813dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:15.102 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ede9e9640 1 -- 172.21.15.153:0/3347342446 shutdown_connections 2024-03-27T03:10:15.102 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.105+0000 7f4ede9e9640 1 -- 172.21.15.153:0/3347342446 >> 172.21.15.153:0/3347342446 conn(0x7f4ed806b660 msgr2=0x7f4ed806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:15.102 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.106+0000 7f4ede9e9640 1 -- 172.21.15.153:0/3347342446 shutdown_connections 2024-03-27T03:10:15.102 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.106+0000 7f4ede9e9640 1 -- 172.21.15.153:0/3347342446 wait complete. 2024-03-27T03:10:15.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.106+0000 7f4ede9e9640 10 monclient: build_initial_monmap 2024-03-27T03:10:15.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.107+0000 7f4ede9e9640 10 monclient: monmap: 2024-03-27T03:10:15.105 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:15.105 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.107+0000 7f4ede9e9640 1 librados: starting msgr at 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.107+0000 7f4ede9e9640 1 librados: starting objecter 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.107+0000 7f4ede9e9640 1 Processor -- start 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.107+0000 7f4ede9e9640 1 -- start start 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.107+0000 7f4ede9e9640 1 librados: setting wanted keys 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.107+0000 7f4ede9e9640 1 librados: calling monclient init 2024-03-27T03:10:15.106 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.107+0000 7f4ede9e9640 10 monclient: init 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.108+0000 7f4ede9e9640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.108+0000 7f4ede9e9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.108+0000 7f4ede9e9640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed8071c30 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.108+0000 7f4ede9e9640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed80a9750 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.108+0000 7f4ede9e9640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed813e1e0 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.108+0000 7f4ede9e9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed811b540 con 0x7f4ed80a9750 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.108+0000 7f4ede9e9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed81d92d0 con 0x7f4ed8071c30 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.108+0000 7f4ede9e9640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed81d8fb0 con 0x7f4ed813e1e0 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.108+0000 7f4ede9e9640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.108+0000 7f4ede9e9640 10 monclient(hunting): authenticate will time out at 1575.414490s 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.108+0000 7f4edcf86640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f4ed813e1e0 0x7f4ed81dfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:42302/0 (socket says 172.21.15.153:42302) 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.108+0000 7f4edcf86640 1 -- 172.21.15.153:0/2303950989 learned_addr learned my addr 172.21.15.153:0/2303950989 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 10 client.?.objecter ms_handle_connect 0x7f4ed813e1e0 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769008743 0 0) 0x7f4ed81d8fb0 con 0x7f4ed813e1e0 2024-03-27T03:10:15.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 10 monclient(hunting): _negotiate 2024-03-27T03:10:15.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:15.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:15.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 10 monclient(hunting): my global_id is 4421 2024-03-27T03:10:15.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb8004eb0 con 0x7f4ed813e1e0 2024-03-27T03:10:15.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2368603195 0 0) 0x7f4eb8004eb0 con 0x7f4ed813e1e0 2024-03-27T03:10:15.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ed81d8fb0 con 0x7f4ed813e1e0 2024-03-27T03:10:15.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4ed0001d30 con 0x7f4ed813e1e0 2024-03-27T03:10:15.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:15.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:10:15.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 10 monclient(hunting): dump: 2024-03-27T03:10:15.109 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:15.109 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:15.109 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:15.109 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:15.109 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:15.109 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:15.109 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:15.109 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:15.109 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:15.109 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:15.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3419344964 0 0) 0x7f4ed81d8fb0 con 0x7f4ed813e1e0 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 >> v1:172.21.15.196:6789/0 conn(0x7f4ed8071c30 legacy=0x7f4ed81d88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.109+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 >> v1:172.21.15.153:6789/0 conn(0x7f4ed80a9750 legacy=0x7f4ed81d53a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ed57fa640 10 monclient: _finish_hunting 0 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ed57fa640 1 monclient: found mon.c 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ed57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ed57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed81421e0 con 0x7f4ed813e1e0 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ed57fa640 10 monclient: _finish_auth 0 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ed57fa640 10 monclient: _check_auth_tickets 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ed57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ed57fa640 10 client.?.objecter ms_handle_connect 0x7f4ed8071c30 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ed57fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ede9e9640 5 monclient: authenticate success, global_id 4421 2024-03-27T03:10:15.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ede9e9640 10 monclient: _renew_subs 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ede9e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ede9e9640 1 -- 172.21.15.153:0/2303950989 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed8142490 con 0x7f4ed813e1e0 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ede9e9640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ede9e9640 10 monclient: _renew_subs 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.110+0000 7f4ede9e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.111+0000 7f4ede9e9640 1 -- 172.21.15.153:0/2303950989 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed81d5c90 con 0x7f4ed813e1e0 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.111+0000 7f4ede9e9640 1 librados: init done 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.111+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ed0002030 con 0x7f4ed813e1e0 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.111+0000 7f4ed57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.111+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4ed0003de0 con 0x7f4ed813e1e0 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.111+0000 7f4ed57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.111+0000 7f4ed57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.111+0000 7f4ed57fa640 10 monclient: dump: 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:15.111 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:15.112 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:15.112 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:15.112 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:15.112 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:15.112 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:15.112 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:15.112 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.111+0000 7f4ed6ffd640 4 set_mon_vals no callback set 2024-03-27T03:10:15.112 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.112+0000 7f4ede9e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:15.112 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.112+0000 7f4ede9e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:15.112 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.112+0000 7f4ede9e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:15.112 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.112+0000 7f4ede9e9640 1 -- 172.21.15.153:0/2303950989 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ebc005080 con 0x7f4ed813e1e0 2024-03-27T03:10:15.112 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.112+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f4ed0003f40 con 0x7f4ed813e1e0 2024-03-27T03:10:15.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.113+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f4ed0084ee0 con 0x7f4ed813e1e0 2024-03-27T03:10:15.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.113+0000 7f4ed57fa640 10 client.4421.objecter ms_dispatch 0x7f4ed8000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:10:15.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.113+0000 7f4ed57fa640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:10:15.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.113+0000 7f4ed57fa640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:10:15.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.113+0000 7f4ed57fa640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-27T03:10:15.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.114+0000 7f4ed7fff640 10 monclient: get_auth_request con 0x7f4eb806dc10 auth_method 0 2024-03-27T03:10:15.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.114+0000 7f4ed57fa640 10 client.4421.objecter ms_handle_connect 0x7f4eb806dc10 2024-03-27T03:10:15.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.119+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ed00507c0 con 0x7f4ed813e1e0 2024-03-27T03:10:15.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.120+0000 7f4ed57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:15.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.120+0000 7f4ed57fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:10:15.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.448+0000 7f4ede9e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:15.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.448+0000 7f4ede9e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:15.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.448+0000 7f4ede9e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:15.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.448+0000 7f4ede9e9640 1 -- 172.21.15.153:0/2303950989 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ebc005330 con 0x7f4ed813e1e0 2024-03-27T03:10:15.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.449+0000 7f4ed57fa640 1 -- 172.21.15.153:0/2303950989 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f4ed00526f0 con 0x7f4ed813e1e0 2024-03-27T03:10:15.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.449+0000 7f4ed57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:15.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.449+0000 7f4ed57fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:10:15.442 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:10:15.443 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:10:15.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.452+0000 7f4ede9e9640 10 librados: watch_flush enter 2024-03-27T03:10:15.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.452+0000 7f4ede9e9640 10 librados: watch_flush exit 2024-03-27T03:10:15.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.452+0000 7f4ede9e9640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-27T03:10:15.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.452+0000 7f4ede9e9640 10 client.4421.objecter successfully canceled tick 2024-03-27T03:10:15.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.453+0000 7f4ede9e9640 1 -- 172.21.15.153:0/2303950989 >> v1:172.21.15.153:6816/2637937781 conn(0x7f4eb806dc10 legacy=0x7f4eb80700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:15.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.453+0000 7f4ede9e9640 10 monclient: shutdown 2024-03-27T03:10:15.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.453+0000 7f4ede9e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:15.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.453+0000 7f4ede9e9640 1 -- 172.21.15.153:0/2303950989 >> v1:172.21.15.153:6790/0 conn(0x7f4ed813e1e0 legacy=0x7f4ed81dfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:15.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.453+0000 7f4ede9e9640 1 -- 172.21.15.153:0/2303950989 shutdown_connections 2024-03-27T03:10:15.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.453+0000 7f4ede9e9640 1 -- 172.21.15.153:0/2303950989 >> 172.21.15.153:0/2303950989 conn(0x7f4ed806b660 msgr2=0x7f4ed806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:15.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.453+0000 7f4ede9e9640 1 -- 172.21.15.153:0/2303950989 shutdown_connections 2024-03-27T03:10:15.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.453+0000 7f4ede9e9640 1 -- 172.21.15.153:0/2303950989 wait complete. 2024-03-27T03:10:15.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:15.453+0000 7f4ede9e9640 1 librados: shutdown 2024-03-27T03:10:15.464 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:10:22.270 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:10:22.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.425+0000 7fc3e4dbc640 10 monclient: get_monmap_and_config 2024-03-27T03:10:22.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.425+0000 7fc3e4dbc640 10 monclient: build_initial_monmap 2024-03-27T03:10:22.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.425+0000 7fc3e4dbc640 10 monclient: monmap: 2024-03-27T03:10:22.419 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:22.419 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:22.419 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:10:22.426125+0000 2024-03-27T03:10:22.419 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:10:22.426125+0000 2024-03-27T03:10:22.419 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:22.419 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:22.419 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:22.419 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:10:22.419 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:10:22.419 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:22.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.427+0000 7fc3e4dbc640 1 Processor -- start 2024-03-27T03:10:22.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.427+0000 7fc3e4dbc640 1 -- start start 2024-03-27T03:10:22.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.427+0000 7fc3e4dbc640 10 monclient: init 2024-03-27T03:10:22.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.428+0000 7fc3e4dbc640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:22.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.428+0000 7fc3e4dbc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:10:22.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.428+0000 7fc3e4dbc640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d80389e0 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:22.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.428+0000 7fc3e4dbc640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d8038dc0 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:22.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.428+0000 7fc3e4dbc640 10 monclient(hunting): picked mon.noname-c con 0x7fc3d8030660 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:22.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.428+0000 7fc3e4dbc640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d80c54c0 con 0x7fc3d8038dc0 2024-03-27T03:10:22.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.428+0000 7fc3e4dbc640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d8036430 con 0x7fc3d8030660 2024-03-27T03:10:22.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.428+0000 7fc3e4dbc640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d8039840 con 0x7fc3d80389e0 2024-03-27T03:10:22.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.428+0000 7fc3e4dbc640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:22.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.428+0000 7fc3e4dbc640 10 monclient(hunting): authenticate will time out at 1582.734468s 2024-03-27T03:10:22.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.429+0000 7fc3ded9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc3d8030660 0x7fc3d80d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:41538/0 (socket says 172.21.15.153:41538) 2024-03-27T03:10:22.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.429+0000 7fc3ded9d640 1 -- 172.21.15.153:0/2647415625 learned_addr learned my addr 172.21.15.153:0/2647415625 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:22.426 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.431+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397519211 0 0) 0x7fc3d8036430 con 0x7fc3d8030660 2024-03-27T03:10:22.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.432+0000 7fc3dd59a640 10 monclient(hunting): _negotiate 2024-03-27T03:10:22.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.432+0000 7fc3dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:22.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.432+0000 7fc3dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:22.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.432+0000 7fc3dd59a640 10 monclient(hunting): my global_id is 4414 2024-03-27T03:10:22.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.432+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c8003520 con 0x7fc3d8030660 2024-03-27T03:10:22.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.432+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850194018 0 0) 0x7fc3d80c54c0 con 0x7fc3d8038dc0 2024-03-27T03:10:22.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.432+0000 7fc3dd59a640 10 monclient(hunting): _negotiate 2024-03-27T03:10:22.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.432+0000 7fc3dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:22.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.432+0000 7fc3dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:22.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.432+0000 7fc3dd59a640 10 monclient(hunting): my global_id is 4395 2024-03-27T03:10:22.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.432+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d8036430 con 0x7fc3d8038dc0 2024-03-27T03:10:22.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269096749 0 0) 0x7fc3d8039840 con 0x7fc3d80389e0 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient(hunting): _negotiate 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient(hunting): my global_id is 4424 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d80c54c0 con 0x7fc3d80389e0 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4118766378 0 0) 0x7fc3d8036430 con 0x7fc3d8038dc0 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 >> v1:172.21.15.153:6790/0 conn(0x7fc3d80389e0 legacy=0x7fc3d8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 >> v1:172.21.15.196:6789/0 conn(0x7fc3d8030660 legacy=0x7fc3d80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient: _finish_hunting 0 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 1 monclient: found mon.noname-a 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d8035690 con 0x7fc3d8038dc0 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient: _finish_auth 0 2024-03-27T03:10:22.428 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient: _check_auth_tickets 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc3d4001670 con 0x7fc3d8038dc0 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient: dump: 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:22.429 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3e4dbc640 5 monclient: authenticate success, global_id 4395 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3e4dbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3d4001890 con 0x7fc3d8038dc0 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 1 -- 172.21.15.153:0/2647415625 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc3d4001c20 con 0x7fc3d8038dc0 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3dd59a640 10 monclient: dump: 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:22.430 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:22.431 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:22.431 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:22.431 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:22.431 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:22.431 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3e4dbc640 10 monclient: get_monmap_and_config success 2024-03-27T03:10:22.431 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.433+0000 7fc3e4dbc640 4 set_mon_vals no callback set 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.434+0000 7fc3e4dbc640 10 monclient: shutdown 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.434+0000 7fc3e4dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.434+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2647415625 >> v1:172.21.15.153:6789/0 conn(0x7fc3d8038dc0 legacy=0x7fc3d80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.435+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2647415625 shutdown_connections 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.435+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2647415625 >> 172.21.15.153:0/2647415625 conn(0x7fc3d8005c00 msgr2=0x7fc3d8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.437+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2647415625 shutdown_connections 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.437+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2647415625 wait complete. 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.438+0000 7fc3e4dbc640 10 monclient: build_initial_monmap 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.438+0000 7fc3e4dbc640 10 monclient: monmap: 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:10:22.433 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.439+0000 7fc3e4dbc640 1 librados: starting msgr at 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.439+0000 7fc3e4dbc640 1 librados: starting objecter 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.439+0000 7fc3e4dbc640 1 Processor -- start 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.439+0000 7fc3e4dbc640 1 -- start start 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.439+0000 7fc3e4dbc640 1 librados: setting wanted keys 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.439+0000 7fc3e4dbc640 1 librados: calling monclient init 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.439+0000 7fc3e4dbc640 10 monclient: init 2024-03-27T03:10:22.434 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.440+0000 7fc3e4dbc640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:22.435 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.440+0000 7fc3e4dbc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:10:22.437 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.442+0000 7fc3e4dbc640 10 monclient(hunting): picked mon.noname-c con 0x7fc3d8030660 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:22.437 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.442+0000 7fc3e4dbc640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d80389e0 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:22.437 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.442+0000 7fc3e4dbc640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d80c8430 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:22.437 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.442+0000 7fc3e4dbc640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d802e620 con 0x7fc3d80c8430 2024-03-27T03:10:22.437 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.442+0000 7fc3e4dbc640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d802e890 con 0x7fc3d80389e0 2024-03-27T03:10:22.437 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.442+0000 7fc3e4dbc640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d815c820 con 0x7fc3d8030660 2024-03-27T03:10:22.437 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.442+0000 7fc3e4dbc640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:22.437 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.442+0000 7fc3e4dbc640 10 monclient(hunting): authenticate will time out at 1582.748151s 2024-03-27T03:10:22.437 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.442+0000 7fc3ddd9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc3d80389e0 0x7fc3d802f6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:45122/0 (socket says 172.21.15.153:45122) 2024-03-27T03:10:22.437 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.442+0000 7fc3ddd9b640 1 -- 172.21.15.153:0/2196989615 learned_addr learned my addr 172.21.15.153:0/2196989615 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.443+0000 7fc3ceffd640 10 client.?.objecter ms_handle_connect 0x7fc3d80389e0 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.443+0000 7fc3ceffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.443+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263759072 0 0) 0x7fc3d802e890 con 0x7fc3d80389e0 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.443+0000 7fc3ceffd640 10 monclient(hunting): _negotiate 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.443+0000 7fc3ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.443+0000 7fc3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.443+0000 7fc3ceffd640 10 monclient(hunting): my global_id is 4417 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.443+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c000d6c0 con 0x7fc3d80389e0 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.444+0000 7fc3ceffd640 10 client.?.objecter ms_handle_connect 0x7fc3d8030660 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.444+0000 7fc3ceffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.444+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029303770 0 0) 0x7fc3d815c820 con 0x7fc3d8030660 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.444+0000 7fc3ceffd640 10 monclient(hunting): _negotiate 2024-03-27T03:10:22.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.444+0000 7fc3ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:22.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.444+0000 7fc3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:22.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.444+0000 7fc3ceffd640 10 monclient(hunting): my global_id is 4427 2024-03-27T03:10:22.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.444+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d802e890 con 0x7fc3d8030660 2024-03-27T03:10:22.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.445+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2651843149 0 0) 0x7fc3d802e890 con 0x7fc3d8030660 2024-03-27T03:10:22.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.445+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3d815c820 con 0x7fc3d8030660 2024-03-27T03:10:22.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.445+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc3e004fdd0 con 0x7fc3d8030660 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.445+0000 7fc3ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.445+0000 7fc3ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.445+0000 7fc3ceffd640 10 monclient(hunting): dump: 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.445+0000 7fc3ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.445+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 404395447 0 0) 0x7fc3d815c820 con 0x7fc3d8030660 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.445+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 >> v1:172.21.15.196:6789/0 conn(0x7fc3d80389e0 legacy=0x7fc3d802f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:22.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.445+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 >> v1:172.21.15.153:6789/0 conn(0x7fc3d80c8430 legacy=0x7fc3d8167f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:10:22.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.448+0000 7fc3ceffd640 10 monclient: _finish_hunting 0 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.448+0000 7fc3ceffd640 1 monclient: found mon.c 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3ceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d80c8150 con 0x7fc3d8030660 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3ceffd640 10 monclient: _finish_auth 0 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3ceffd640 10 monclient: _check_auth_tickets 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3e4dbc640 5 monclient: authenticate success, global_id 4427 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3e4dbc640 10 monclient: _renew_subs 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3e4dbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2196989615 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d815d450 con 0x7fc3d8030660 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3e4dbc640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3e4dbc640 10 monclient: _renew_subs 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3e4dbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:22.442 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2196989615 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d802f170 con 0x7fc3d8030660 2024-03-27T03:10:22.443 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.449+0000 7fc3e4dbc640 1 librados: init done 2024-03-27T03:10:22.444 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.451+0000 7fc3e4dbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:22.444 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.451+0000 7fc3e4dbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:22.444 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.451+0000 7fc3e4dbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:22.444 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.451+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2196989615 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d8007b90 con 0x7fc3d8030660 2024-03-27T03:10:22.445 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.452+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3e004f400 con 0x7fc3d8030660 2024-03-27T03:10:22.445 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.452+0000 7fc3ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:22.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.453+0000 7fc3dcd99640 4 set_mon_vals no callback set 2024-03-27T03:10:22.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.453+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc3e0075f10 con 0x7fc3d8030660 2024-03-27T03:10:22.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.453+0000 7fc3ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:22.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.453+0000 7fc3ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:10:22.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.453+0000 7fc3ceffd640 10 monclient: dump: 2024-03-27T03:10:22.447 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:22.447 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:22.447 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:22.447 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:22.447 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:22.447 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:22.448 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:22.448 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:22.448 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:22.448 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:22.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.453+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fc3e0067bf0 con 0x7fc3d8030660 2024-03-27T03:10:22.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.454+0000 7fc3ddd9b640 10 monclient: get_auth_request con 0x7fc3c0076390 auth_method 0 2024-03-27T03:10:22.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.454+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fc3e00fe410 con 0x7fc3d8030660 2024-03-27T03:10:22.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.454+0000 7fc3ceffd640 10 client.4427.objecter ms_dispatch 0x7fc3e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:10:22.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.454+0000 7fc3ceffd640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:10:22.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.454+0000 7fc3ceffd640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:10:22.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.454+0000 7fc3ceffd640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-27T03:10:22.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.454+0000 7fc3ceffd640 10 client.4427.objecter ms_handle_connect 0x7fc3c0076390 2024-03-27T03:10:22.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.462+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3e00c9e30 con 0x7fc3d8030660 2024-03-27T03:10:22.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.462+0000 7fc3ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:22.455 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.462+0000 7fc3ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:10:22.788 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.795+0000 7fc3e4dbc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:22.788 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.795+0000 7fc3e4dbc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:22.788 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.795+0000 7fc3e4dbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:22.788 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.795+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2196989615 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3d8005120 con 0x7fc3d8030660 2024-03-27T03:10:22.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.796+0000 7fc3ceffd640 1 -- 172.21.15.153:0/2196989615 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fc3e0067f20 con 0x7fc3d8030660 2024-03-27T03:10:22.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.796+0000 7fc3ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:22.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.796+0000 7fc3ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:10:22.790 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:10:22.790 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:10:22.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.800+0000 7fc3e4dbc640 10 librados: watch_flush enter 2024-03-27T03:10:22.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.800+0000 7fc3e4dbc640 10 librados: watch_flush exit 2024-03-27T03:10:22.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.800+0000 7fc3e4dbc640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-27T03:10:22.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.800+0000 7fc3e4dbc640 10 client.4427.objecter successfully canceled tick 2024-03-27T03:10:22.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.800+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2196989615 >> v1:172.21.15.153:6816/2637937781 conn(0x7fc3c0076390 legacy=0x7fc3c0078850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:22.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.800+0000 7fc3e4dbc640 10 monclient: shutdown 2024-03-27T03:10:22.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.800+0000 7fc3e4dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:22.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.800+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2196989615 >> v1:172.21.15.153:6790/0 conn(0x7fc3d8030660 legacy=0x7fc3d815cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:22.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.801+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2196989615 shutdown_connections 2024-03-27T03:10:22.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.801+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2196989615 >> 172.21.15.153:0/2196989615 conn(0x7fc3d8005c00 msgr2=0x7fc3d8034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:22.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.801+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2196989615 shutdown_connections 2024-03-27T03:10:22.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.801+0000 7fc3e4dbc640 1 -- 172.21.15.153:0/2196989615 wait complete. 2024-03-27T03:10:22.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:22.801+0000 7fc3e4dbc640 1 librados: shutdown 2024-03-27T03:10:22.812 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:10:29.619 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:10:29.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.772+0000 7faf4e002640 10 monclient: get_monmap_and_config 2024-03-27T03:10:29.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.772+0000 7faf4e002640 10 monclient: build_initial_monmap 2024-03-27T03:10:29.766 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.772+0000 7faf4e002640 10 monclient: monmap: 2024-03-27T03:10:29.766 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:29.766 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:29.766 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:10:29.773594+0000 2024-03-27T03:10:29.766 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:10:29.773594+0000 2024-03-27T03:10:29.766 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:29.766 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:29.766 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:29.766 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:10:29.766 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:10:29.766 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:29.770 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.777+0000 7faf4e002640 1 Processor -- start 2024-03-27T03:10:29.770 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.777+0000 7faf4e002640 1 -- start start 2024-03-27T03:10:29.770 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.777+0000 7faf4e002640 10 monclient: init 2024-03-27T03:10:29.771 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.778+0000 7faf4e002640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:29.771 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.778+0000 7faf4e002640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:10:29.771 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.778+0000 7faf4e002640 10 monclient(hunting): picked mon.noname-c con 0x7faf481499d0 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:29.782 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.782+0000 7faf4e002640 10 monclient(hunting): picked mon.noname-b con 0x7faf480a9fc0 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:29.782 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.782+0000 7faf4e002640 10 monclient(hunting): picked mon.noname-a con 0x7faf4814faf0 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:29.782 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.782+0000 7faf4e002640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf480abd70 con 0x7faf4814faf0 2024-03-27T03:10:29.782 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.782+0000 7faf4e002640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf480a9850 con 0x7faf481499d0 2024-03-27T03:10:29.782 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.782+0000 7faf47fff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7faf480a9fc0 0x7faf480aa420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:34319/0 (socket says 172.21.15.153:34319) 2024-03-27T03:10:29.782 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.782+0000 7faf47fff640 1 -- 172.21.15.153:0/2606891802 learned_addr learned my addr 172.21.15.153:0/2606891802 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:29.782 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.785+0000 7faf4e002640 1 -- 172.21.15.153:0/2606891802 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf481405c0 con 0x7faf480a9fc0 2024-03-27T03:10:29.782 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.785+0000 7faf4e002640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:29.782 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.785+0000 7faf4e002640 10 monclient(hunting): authenticate will time out at 1590.090640s 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.785+0000 7faf477fe640 1 -- 172.21.15.153:0/2606891802 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823234376 0 0) 0x7faf481405c0 con 0x7faf480a9fc0 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.785+0000 7faf477fe640 10 monclient(hunting): _negotiate 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.785+0000 7faf477fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.785+0000 7faf477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.785+0000 7faf477fe640 10 monclient(hunting): my global_id is 4430 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.785+0000 7faf477fe640 1 -- 172.21.15.153:0/2606891802 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf30003520 con 0x7faf480a9fc0 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 1 -- 172.21.15.153:0/2606891802 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 30605315 0 0) 0x7faf30003520 con 0x7faf480a9fc0 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 1 -- 172.21.15.153:0/2606891802 >> v1:172.21.15.196:6789/0 conn(0x7faf481499d0 legacy=0x7faf480ad820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 1 -- 172.21.15.153:0/2606891802 >> v1:172.21.15.153:6789/0 conn(0x7faf4814faf0 legacy=0x7faf48151ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 10 monclient: _finish_hunting 0 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 1 monclient: found mon.noname-b 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 1 -- 172.21.15.153:0/2606891802 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf48143d20 con 0x7faf480a9fc0 2024-03-27T03:10:29.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 10 monclient: _finish_auth 0 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 10 monclient: _check_auth_tickets 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 1 -- 172.21.15.153:0/2606891802 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7faf40001a10 con 0x7faf480a9fc0 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf477fe640 10 monclient: dump: 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:29.784 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf4e002640 5 monclient: authenticate success, global_id 4430 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.786+0000 7faf4e002640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.787+0000 7faf477fe640 1 -- 172.21.15.153:0/2606891802 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf40001be0 con 0x7faf480a9fc0 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.787+0000 7faf477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.787+0000 7faf477fe640 1 -- 172.21.15.153:0/2606891802 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7faf40002030 con 0x7faf480a9fc0 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.787+0000 7faf477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.787+0000 7faf477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.787+0000 7faf477fe640 10 monclient: dump: 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:29.785 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:29.786 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:29.786 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:29.786 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:29.786 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:29.786 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.787+0000 7faf4e002640 10 monclient: get_monmap_and_config success 2024-03-27T03:10:29.786 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.787+0000 7faf4e002640 4 set_mon_vals no callback set 2024-03-27T03:10:29.786 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.788+0000 7faf4e002640 10 monclient: shutdown 2024-03-27T03:10:29.786 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.788+0000 7faf4e002640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:29.786 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.788+0000 7faf4e002640 1 -- 172.21.15.153:0/2606891802 >> v1:172.21.15.153:6790/0 conn(0x7faf480a9fc0 legacy=0x7faf480aa420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:29.786 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.789+0000 7faf4e002640 1 -- 172.21.15.153:0/2606891802 shutdown_connections 2024-03-27T03:10:29.786 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.789+0000 7faf4e002640 1 -- 172.21.15.153:0/2606891802 >> 172.21.15.153:0/2606891802 conn(0x7faf480b3c50 msgr2=0x7faf480b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:29.786 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.789+0000 7faf4e002640 1 -- 172.21.15.153:0/2606891802 shutdown_connections 2024-03-27T03:10:29.788 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.789+0000 7faf4e002640 1 -- 172.21.15.153:0/2606891802 wait complete. 2024-03-27T03:10:29.788 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.790+0000 7faf4e002640 10 monclient: build_initial_monmap 2024-03-27T03:10:29.788 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.790+0000 7faf4e002640 10 monclient: monmap: 2024-03-27T03:10:29.788 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:29.788 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:10:29.788 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:10:29.788 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:10:29.788 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:29.788 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:29.788 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.790+0000 7faf4e002640 1 librados: starting msgr at 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.790+0000 7faf4e002640 1 librados: starting objecter 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.791+0000 7faf4e002640 1 Processor -- start 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.791+0000 7faf4e002640 1 -- start start 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.791+0000 7faf4e002640 1 librados: setting wanted keys 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.791+0000 7faf4e002640 1 librados: calling monclient init 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.791+0000 7faf4e002640 10 monclient: init 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.792+0000 7faf4e002640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.792+0000 7faf4e002640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:10:29.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.792+0000 7faf4e002640 10 monclient(hunting): picked mon.noname-c con 0x7faf480a9fc0 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:29.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.795+0000 7faf4d000640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7faf480a9fc0 0x7faf48142a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:34433/0 (socket says 172.21.15.153:34433) 2024-03-27T03:10:29.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.797+0000 7faf4e002640 10 monclient(hunting): picked mon.noname-b con 0x7faf4814faf0 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:29.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.798+0000 7faf4e002640 10 monclient(hunting): picked mon.noname-a con 0x7faf481ded60 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:29.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.798+0000 7faf4d000640 1 -- 172.21.15.153:0/2557147914 learned_addr learned my addr 172.21.15.153:0/2557147914 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:29.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.798+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4813c1d0 con 0x7faf481ded60 2024-03-27T03:10:29.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.798+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4806e2e0 con 0x7faf4814faf0 2024-03-27T03:10:29.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.798+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4806e5c0 con 0x7faf480a9fc0 2024-03-27T03:10:29.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.798+0000 7faf4e002640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:29.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.798+0000 7faf4e002640 10 monclient(hunting): authenticate will time out at 1590.104220s 2024-03-27T03:10:29.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 10 client.?.objecter ms_handle_connect 0x7faf480a9fc0 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889815189 0 0) 0x7faf4806e5c0 con 0x7faf480a9fc0 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 10 monclient(hunting): _negotiate 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 10 monclient(hunting): my global_id is 4433 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf2c004eb0 con 0x7faf480a9fc0 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 10 client.?.objecter ms_handle_connect 0x7faf481ded60 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274993654 0 0) 0x7faf4813c1d0 con 0x7faf481ded60 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 10 monclient(hunting): _negotiate 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:29.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 10 monclient(hunting): my global_id is 4398 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.802+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf4806e5c0 con 0x7faf481ded60 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3422605535 0 0) 0x7faf2c004eb0 con 0x7faf480a9fc0 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf4813c1d0 con 0x7faf480a9fc0 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7faf3c003ff0 con 0x7faf480a9fc0 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 10 monclient(hunting): dump: 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:29.802 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1370724785 0 0) 0x7faf4806e5c0 con 0x7faf481ded60 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf2c004eb0 con 0x7faf481ded60 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7faf38001710 con 0x7faf481ded60 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.803+0000 7faf457fa640 10 monclient(hunting): dump: 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:29.803 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.806+0000 7faf457fa640 10 client.?.objecter ms_handle_connect 0x7faf4814faf0 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.806+0000 7faf457fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.806+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3878634968 0 0) 0x7faf2c004eb0 con 0x7faf481ded60 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.806+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 >> v1:172.21.15.153:6790/0 conn(0x7faf480a9fc0 legacy=0x7faf48142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.806+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 >> v1:172.21.15.196:6789/0 conn(0x7faf4814faf0 legacy=0x7faf4813b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.806+0000 7faf457fa640 10 monclient: _finish_hunting 0 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.806+0000 7faf457fa640 1 monclient: found mon.a 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.806+0000 7faf457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.806+0000 7faf457fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:10:29.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.806+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf48143150 con 0x7faf481ded60 2024-03-27T03:10:29.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.806+0000 7faf457fa640 10 monclient: _finish_auth 0 2024-03-27T03:10:29.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.807+0000 7faf457fa640 10 monclient: _check_auth_tickets 2024-03-27T03:10:29.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.807+0000 7faf457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:29.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.808+0000 7faf4e002640 5 monclient: authenticate success, global_id 4398 2024-03-27T03:10:29.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.808+0000 7faf4e002640 10 monclient: _renew_subs 2024-03-27T03:10:29.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.808+0000 7faf4e002640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:10:29.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.808+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf48146010 con 0x7faf481ded60 2024-03-27T03:10:29.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.808+0000 7faf4e002640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:10:29.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.808+0000 7faf4e002640 10 monclient: _renew_subs 2024-03-27T03:10:29.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.808+0000 7faf4e002640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.808+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf48141f30 con 0x7faf481ded60 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.808+0000 7faf4e002640 1 librados: init done 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.810+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf38001870 con 0x7faf481ded60 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.810+0000 7faf457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.810+0000 7faf46ffd640 4 set_mon_vals no callback set 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.811+0000 7faf4e002640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.811+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7faf38003790 con 0x7faf481ded60 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.811+0000 7faf457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.811+0000 7faf457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.811+0000 7faf457fa640 10 monclient: dump: 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:29.807 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:29.808 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:29.808 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:29.808 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:29.808 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:29.808 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:29.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.811+0000 7faf4e002640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:29.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.811+0000 7faf4e002640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:10:29.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.811+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf24005080 con 0x7faf481ded60 2024-03-27T03:10:29.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.815+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7faf38019760 con 0x7faf481ded60 2024-03-27T03:10:29.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.816+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7faf38083600 con 0x7faf481ded60 2024-03-27T03:10:29.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.816+0000 7faf457fa640 10 client.4398.objecter ms_dispatch 0x7faf48000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:10:29.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.816+0000 7faf457fa640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:10:29.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.816+0000 7faf457fa640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:10:29.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.816+0000 7faf457fa640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-27T03:10:29.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.817+0000 7faf4d000640 10 monclient: get_auth_request con 0x7faf2c06def0 auth_method 0 2024-03-27T03:10:29.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.817+0000 7faf457fa640 10 client.4398.objecter ms_handle_connect 0x7faf2c06def0 2024-03-27T03:10:29.816 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.822+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf3804eee0 con 0x7faf481ded60 2024-03-27T03:10:29.816 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.822+0000 7faf457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:29.816 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:29.822+0000 7faf457fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:10:30.150 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.157+0000 7faf4e002640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:30.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.157+0000 7faf4e002640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:30.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.157+0000 7faf4e002640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:10:30.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.157+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf24005330 con 0x7faf481ded60 2024-03-27T03:10:30.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.163+0000 7faf457fa640 1 -- 172.21.15.153:0/2557147914 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7faf38050e10 con 0x7faf481ded60 2024-03-27T03:10:30.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.163+0000 7faf457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:30.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.163+0000 7faf457fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:10:30.159 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:10:30.159 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:10:30.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.168+0000 7faf4e002640 10 librados: watch_flush enter 2024-03-27T03:10:30.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.168+0000 7faf4e002640 10 librados: watch_flush exit 2024-03-27T03:10:30.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.168+0000 7faf4e002640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-27T03:10:30.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.168+0000 7faf4e002640 10 client.4398.objecter successfully canceled tick 2024-03-27T03:10:30.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.168+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 >> v1:172.21.15.153:6816/2637937781 conn(0x7faf2c06def0 legacy=0x7faf2c0703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:30.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.168+0000 7faf4e002640 10 monclient: shutdown 2024-03-27T03:10:30.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.168+0000 7faf4e002640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:30.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.168+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 >> v1:172.21.15.153:6789/0 conn(0x7faf481ded60 legacy=0x7faf4813bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:30.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.171+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 shutdown_connections 2024-03-27T03:10:30.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.171+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 >> 172.21.15.153:0/2557147914 conn(0x7faf480b3c50 msgr2=0x7faf480aa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:30.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.172+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 shutdown_connections 2024-03-27T03:10:30.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.172+0000 7faf4e002640 1 -- 172.21.15.153:0/2557147914 wait complete. 2024-03-27T03:10:30.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:30.172+0000 7faf4e002640 1 librados: shutdown 2024-03-27T03:10:30.187 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:10:36.992 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:10:37.144 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.150+0000 7f831f932640 10 monclient: get_monmap_and_config 2024-03-27T03:10:37.144 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.150+0000 7f831f932640 10 monclient: build_initial_monmap 2024-03-27T03:10:37.144 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.150+0000 7f831f932640 10 monclient: monmap: 2024-03-27T03:10:37.144 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:37.144 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:37.144 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:10:37.151789+0000 2024-03-27T03:10:37.144 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:10:37.151789+0000 2024-03-27T03:10:37.144 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:37.144 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:37.144 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:37.144 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:10:37.144 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:10:37.145 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:37.145 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.152+0000 7f831f932640 1 Processor -- start 2024-03-27T03:10:37.145 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.152+0000 7f831f932640 1 -- start start 2024-03-27T03:10:37.145 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.152+0000 7f831f932640 10 monclient: init 2024-03-27T03:10:37.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.153+0000 7f831f932640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:37.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.153+0000 7f831f932640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:10:37.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.153+0000 7f831f932640 10 monclient(hunting): picked mon.noname-a con 0x7f83181457f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:37.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.153+0000 7f831f932640 10 monclient(hunting): picked mon.noname-b con 0x7f831813f650 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:37.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.153+0000 7f831f932640 10 monclient(hunting): picked mon.noname-c con 0x7f83180a9d70 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:37.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.153+0000 7f831f932640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83180aaef0 con 0x7f83181457f0 2024-03-27T03:10:37.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.153+0000 7f831f932640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8318145bd0 con 0x7f83180a9d70 2024-03-27T03:10:37.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.153+0000 7f831f932640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8318145eb0 con 0x7f831813f650 2024-03-27T03:10:37.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.153+0000 7f831f932640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:37.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.153+0000 7f831f932640 10 monclient(hunting): authenticate will time out at 1597.459089s 2024-03-27T03:10:37.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.158+0000 7f831cecd640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f831813f650 0x7f83180b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:43927/0 (socket says 172.21.15.153:43927) 2024-03-27T03:10:37.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.158+0000 7f831cecd640 1 -- 172.21.15.153:0/1926071884 learned_addr learned my addr 172.21.15.153:0/1926071884 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:37.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.162+0000 7f830ffff640 1 -- 172.21.15.153:0/1926071884 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689104086 0 0) 0x7f8318145eb0 con 0x7f831813f650 2024-03-27T03:10:37.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.162+0000 7f830ffff640 10 monclient(hunting): _negotiate 2024-03-27T03:10:37.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.162+0000 7f830ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:37.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.162+0000 7f830ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:37.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.162+0000 7f830ffff640 10 monclient(hunting): my global_id is 4436 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.162+0000 7f830ffff640 1 -- 172.21.15.153:0/1926071884 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8304003520 con 0x7f831813f650 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.162+0000 7f830ffff640 1 -- 172.21.15.153:0/1926071884 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017730592 0 0) 0x7f8318145bd0 con 0x7f83180a9d70 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.162+0000 7f830ffff640 10 monclient(hunting): _negotiate 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.162+0000 7f830ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.162+0000 7f830ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.162+0000 7f830ffff640 10 monclient(hunting): my global_id is 4420 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 1 -- 172.21.15.153:0/1926071884 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8318145eb0 con 0x7f83180a9d70 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 1 -- 172.21.15.153:0/1926071884 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2045688140 0 0) 0x7f8304003520 con 0x7f831813f650 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 1 -- 172.21.15.153:0/1926071884 >> v1:172.21.15.196:6789/0 conn(0x7f83180a9d70 legacy=0x7f831814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 1 -- 172.21.15.153:0/1926071884 >> v1:172.21.15.153:6789/0 conn(0x7f83181457f0 legacy=0x7f831813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 10 monclient: _finish_hunting 0 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 1 monclient: found mon.noname-b 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:10:37.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 1 -- 172.21.15.153:0/1926071884 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8318146340 con 0x7f831813f650 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 10 monclient: _finish_auth 0 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 10 monclient: _check_auth_tickets 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 1 -- 172.21.15.153:0/1926071884 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8310001a10 con 0x7f831813f650 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f830ffff640 10 monclient: dump: 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:37.160 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:37.161 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:37.161 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:37.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f831f932640 5 monclient: authenticate success, global_id 4436 2024-03-27T03:10:37.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.163+0000 7f831f932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:10:37.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.166+0000 7f830ffff640 1 -- 172.21.15.153:0/1926071884 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8310001b70 con 0x7f831813f650 2024-03-27T03:10:37.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.166+0000 7f830ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:37.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.166+0000 7f830ffff640 1 -- 172.21.15.153:0/1926071884 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8310001f80 con 0x7f831813f650 2024-03-27T03:10:37.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.166+0000 7f830ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.166+0000 7f830ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.166+0000 7f830ffff640 10 monclient: dump: 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.168+0000 7f831f932640 10 monclient: get_monmap_and_config success 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.168+0000 7f831f932640 4 set_mon_vals no callback set 2024-03-27T03:10:37.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.169+0000 7f831f932640 10 monclient: shutdown 2024-03-27T03:10:37.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.169+0000 7f831f932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:37.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.169+0000 7f831f932640 1 -- 172.21.15.153:0/1926071884 >> v1:172.21.15.153:6790/0 conn(0x7f831813f650 legacy=0x7f83180b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:37.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.169+0000 7f831f932640 1 -- 172.21.15.153:0/1926071884 shutdown_connections 2024-03-27T03:10:37.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.169+0000 7f831f932640 1 -- 172.21.15.153:0/1926071884 >> 172.21.15.153:0/1926071884 conn(0x7f8318066a00 msgr2=0x7f8318066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:37.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.172+0000 7f831f932640 1 -- 172.21.15.153:0/1926071884 shutdown_connections 2024-03-27T03:10:37.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.172+0000 7f831f932640 1 -- 172.21.15.153:0/1926071884 wait complete. 2024-03-27T03:10:37.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.173+0000 7f831f932640 10 monclient: build_initial_monmap 2024-03-27T03:10:37.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.173+0000 7f831f932640 10 monclient: monmap: 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.174+0000 7f831f932640 1 librados: starting msgr at 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.174+0000 7f831f932640 1 librados: starting objecter 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.174+0000 7f831f932640 1 Processor -- start 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.174+0000 7f831f932640 1 -- start start 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.174+0000 7f831f932640 1 librados: setting wanted keys 2024-03-27T03:10:37.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.174+0000 7f831f932640 1 librados: calling monclient init 2024-03-27T03:10:37.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.174+0000 7f831f932640 10 monclient: init 2024-03-27T03:10:37.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.175+0000 7f831f932640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:37.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.175+0000 7f831f932640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:10:37.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.175+0000 7f831f932640 10 monclient(hunting): picked mon.noname-a con 0x7f83180a9d70 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:37.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.182+0000 7f831f932640 10 monclient(hunting): picked mon.noname-c con 0x7f831813f650 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:37.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.182+0000 7f831cecd640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f831813f650 0x7f83181cd4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:44089/0 (socket says 172.21.15.153:44089) 2024-03-27T03:10:37.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.182+0000 7f831cecd640 1 -- 172.21.15.153:0/4132964238 learned_addr learned my addr 172.21.15.153:0/4132964238 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:37.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f831f932640 10 monclient(hunting): picked mon.noname-b con 0x7f83181457f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:37.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83181425d0 con 0x7f83180a9d70 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83181cce40 con 0x7f83181457f0 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83181cd0b0 con 0x7f831813f650 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f831f932640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f831f932640 10 monclient(hunting): authenticate will time out at 1597.488963s 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f830dffb640 10 client.?.objecter ms_handle_connect 0x7f831813f650 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f830dffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f830dffb640 10 client.?.objecter ms_handle_connect 0x7f83180a9d70 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f830dffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549376005 0 0) 0x7f83181cd0b0 con 0x7f831813f650 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f830dffb640 10 monclient(hunting): _negotiate 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.183+0000 7f830dffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 10 monclient(hunting): my global_id is 4439 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82fc004eb0 con 0x7f831813f650 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201714903 0 0) 0x7f83181425d0 con 0x7f83180a9d70 2024-03-27T03:10:37.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 10 monclient(hunting): _negotiate 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 10 monclient(hunting): my global_id is 4401 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83181cd0b0 con 0x7f83180a9d70 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3891532142 0 0) 0x7f82fc004eb0 con 0x7f831813f650 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83181425d0 con 0x7f831813f650 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8310001f40 con 0x7f831813f650 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.184+0000 7f830dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.185+0000 7f830dffb640 10 monclient(hunting): dump: 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:37.180 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:37.181 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:37.181 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:37.181 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:37.181 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:37.181 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:37.181 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.185+0000 7f830dffb640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:10:37.181 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.185+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 647611827 0 0) 0x7f83181425d0 con 0x7f831813f650 2024-03-27T03:10:37.181 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.185+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 >> v1:172.21.15.196:6789/0 conn(0x7f83181457f0 legacy=0x7f83181e1e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:10:37.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.187+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 >> v1:172.21.15.153:6789/0 conn(0x7f83180a9d70 legacy=0x7f83181cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:37.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.187+0000 7f830dffb640 10 monclient: _finish_hunting 0 2024-03-27T03:10:37.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.187+0000 7f830dffb640 1 monclient: found mon.c 2024-03-27T03:10:37.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.187+0000 7f830dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:37.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.187+0000 7f830dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:37.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83181cbcc0 con 0x7f831813f650 2024-03-27T03:10:37.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f830dffb640 10 monclient: _finish_auth 0 2024-03-27T03:10:37.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f830dffb640 10 monclient: _check_auth_tickets 2024-03-27T03:10:37.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f830dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:37.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f831f932640 5 monclient: authenticate success, global_id 4439 2024-03-27T03:10:37.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f831f932640 10 monclient: _renew_subs 2024-03-27T03:10:37.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f831f932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:37.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8318142a60 con 0x7f831813f650 2024-03-27T03:10:37.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f831f932640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:10:37.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f831f932640 10 monclient: _renew_subs 2024-03-27T03:10:37.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f831f932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:37.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83181cdbe0 con 0x7f831813f650 2024-03-27T03:10:37.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.188+0000 7f831f932640 1 librados: init done 2024-03-27T03:10:37.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.190+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8310001570 con 0x7f831813f650 2024-03-27T03:10:37.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.190+0000 7f830dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.190+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8310004fb0 con 0x7f831813f650 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.190+0000 7f830dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.190+0000 7f830dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.190+0000 7f830dffb640 10 monclient: dump: 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:37.186 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:37.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.190+0000 7f830f7fe640 4 set_mon_vals no callback set 2024-03-27T03:10:37.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.191+0000 7f831f932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:37.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.191+0000 7f831f932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:37.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.191+0000 7f831f932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:37.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.191+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f8005080 con 0x7f831813f650 2024-03-27T03:10:37.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.192+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f83100051d0 con 0x7f831813f650 2024-03-27T03:10:37.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.193+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f8310084fe0 con 0x7f831813f650 2024-03-27T03:10:37.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.193+0000 7f830dffb640 10 client.4439.objecter ms_dispatch 0x7f8318000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:10:37.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.193+0000 7f831d6ce640 10 monclient: get_auth_request con 0x7f82fc06daf0 auth_method 0 2024-03-27T03:10:37.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.194+0000 7f830dffb640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:10:37.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.194+0000 7f830dffb640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:10:37.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.194+0000 7f830dffb640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-27T03:10:37.188 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.194+0000 7f830dffb640 10 client.4439.objecter ms_handle_connect 0x7f82fc06daf0 2024-03-27T03:10:37.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.199+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f83100508c0 con 0x7f831813f650 2024-03-27T03:10:37.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.200+0000 7f830dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:37.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.200+0000 7f830dffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:10:37.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.522+0000 7f831f932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:37.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.522+0000 7f831f932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:37.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.522+0000 7f831f932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:37.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.522+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82f8005330 con 0x7f831813f650 2024-03-27T03:10:37.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.523+0000 7f830dffb640 1 -- 172.21.15.153:0/4132964238 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f83100527f0 con 0x7f831813f650 2024-03-27T03:10:37.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.523+0000 7f830dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:37.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.523+0000 7f830dffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:10:37.518 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:10:37.518 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:10:37.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.527+0000 7f831f932640 10 librados: watch_flush enter 2024-03-27T03:10:37.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.527+0000 7f831f932640 10 librados: watch_flush exit 2024-03-27T03:10:37.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.527+0000 7f831f932640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-27T03:10:37.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.527+0000 7f831f932640 10 client.4439.objecter successfully canceled tick 2024-03-27T03:10:37.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.527+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 >> v1:172.21.15.153:6816/2637937781 conn(0x7f82fc06daf0 legacy=0x7f82fc06ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:37.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.527+0000 7f831f932640 10 monclient: shutdown 2024-03-27T03:10:37.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.527+0000 7f831f932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:37.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.527+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 >> v1:172.21.15.153:6790/0 conn(0x7f831813f650 legacy=0x7f83181cd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:37.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.529+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 shutdown_connections 2024-03-27T03:10:37.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.529+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 >> 172.21.15.153:0/4132964238 conn(0x7f8318066a00 msgr2=0x7f83180b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:37.522 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.529+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 shutdown_connections 2024-03-27T03:10:37.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.529+0000 7f831f932640 1 -- 172.21.15.153:0/4132964238 wait complete. 2024-03-27T03:10:37.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:37.530+0000 7f831f932640 1 librados: shutdown 2024-03-27T03:10:37.543 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:10:44.351 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:10:44.501 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.507+0000 7f8d7f08d640 10 monclient: get_monmap_and_config 2024-03-27T03:10:44.501 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.507+0000 7f8d7f08d640 10 monclient: build_initial_monmap 2024-03-27T03:10:44.501 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.507+0000 7f8d7f08d640 10 monclient: monmap: 2024-03-27T03:10:44.502 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:44.502 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:44.502 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:10:44.509401+0000 2024-03-27T03:10:44.502 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:10:44.509401+0000 2024-03-27T03:10:44.502 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:44.502 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:44.502 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:44.502 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:10:44.502 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:10:44.502 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:44.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.509+0000 7f8d7f08d640 1 Processor -- start 2024-03-27T03:10:44.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.509+0000 7f8d7f08d640 1 -- start start 2024-03-27T03:10:44.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.509+0000 7f8d7f08d640 10 monclient: init 2024-03-27T03:10:44.504 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.509+0000 7f8d7f08d640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:44.504 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.509+0000 7f8d7f08d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:10:44.504 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.509+0000 7f8d7f08d640 10 monclient(hunting): picked mon.noname-b con 0x7f8d780b0770 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:44.504 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.509+0000 7f8d7f08d640 10 monclient(hunting): picked mon.noname-c con 0x7f8d7814a700 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:44.510 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.516+0000 7f8d7ce29640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f8d780b0770 0x7f8d78149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:54929/0 (socket says 172.21.15.153:54929) 2024-03-27T03:10:44.511 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.517+0000 7f8d7f08d640 10 monclient(hunting): picked mon.noname-a con 0x7f8d780ac5a0 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:44.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.517+0000 7f8d7ce29640 1 -- 172.21.15.153:0/1532094586 learned_addr learned my addr 172.21.15.153:0/1532094586 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:44.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.517+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1532094586 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d780aba70 con 0x7f8d780ac5a0 2024-03-27T03:10:44.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.517+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1532094586 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d780aecb0 con 0x7f8d7814a700 2024-03-27T03:10:44.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.517+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1532094586 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d780af150 con 0x7f8d780b0770 2024-03-27T03:10:44.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.517+0000 7f8d7f08d640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:44.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.517+0000 7f8d7f08d640 10 monclient(hunting): authenticate will time out at 1604.823957s 2024-03-27T03:10:44.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.517+0000 7f8d777fe640 1 -- 172.21.15.153:0/1532094586 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732172158 0 0) 0x7f8d780af150 con 0x7f8d780b0770 2024-03-27T03:10:44.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.518+0000 7f8d777fe640 10 monclient(hunting): _negotiate 2024-03-27T03:10:44.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.518+0000 7f8d777fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:44.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.518+0000 7f8d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:44.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.518+0000 7f8d777fe640 10 monclient(hunting): my global_id is 4442 2024-03-27T03:10:44.512 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.518+0000 7f8d777fe640 1 -- 172.21.15.153:0/1532094586 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d60003520 con 0x7f8d780b0770 2024-03-27T03:10:44.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.518+0000 7f8d777fe640 1 -- 172.21.15.153:0/1532094586 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853772987 0 0) 0x7f8d780aecb0 con 0x7f8d7814a700 2024-03-27T03:10:44.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.518+0000 7f8d777fe640 10 monclient(hunting): _negotiate 2024-03-27T03:10:44.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.518+0000 7f8d777fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:44.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.518+0000 7f8d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:44.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.518+0000 7f8d777fe640 10 monclient(hunting): my global_id is 4423 2024-03-27T03:10:44.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.518+0000 7f8d777fe640 1 -- 172.21.15.153:0/1532094586 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d780af150 con 0x7f8d7814a700 2024-03-27T03:10:44.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 1 -- 172.21.15.153:0/1532094586 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1981155454 0 0) 0x7f8d780af150 con 0x7f8d7814a700 2024-03-27T03:10:44.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 1 -- 172.21.15.153:0/1532094586 >> v1:172.21.15.153:6790/0 conn(0x7f8d780b0770 legacy=0x7f8d78149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:44.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 1 -- 172.21.15.153:0/1532094586 >> v1:172.21.15.153:6789/0 conn(0x7f8d780ac5a0 legacy=0x7f8d78151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:44.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 10 monclient: _finish_hunting 0 2024-03-27T03:10:44.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 1 monclient: found mon.noname-c 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 1 -- 172.21.15.153:0/1532094586 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d780ac9d0 con 0x7f8d7814a700 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 10 monclient: _finish_auth 0 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 10 monclient: _check_auth_tickets 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d7f08d640 5 monclient: authenticate success, global_id 4423 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d7f08d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 1 -- 172.21.15.153:0/1532094586 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8d6c0015f0 con 0x7f8d7814a700 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d777fe640 10 monclient: dump: 2024-03-27T03:10:44.514 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:44.515 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:44.515 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:44.515 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:44.515 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:44.515 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:44.515 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:44.515 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:44.515 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:44.515 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:44.515 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.519+0000 7f8d7f08d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:10:44.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.520+0000 7f8d777fe640 1 -- 172.21.15.153:0/1532094586 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d6c0017d0 con 0x7f8d7814a700 2024-03-27T03:10:44.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.520+0000 7f8d777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:44.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.520+0000 7f8d7f08d640 10 monclient: get_monmap_and_config success 2024-03-27T03:10:44.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.520+0000 7f8d7f08d640 4 set_mon_vals no callback set 2024-03-27T03:10:44.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.520+0000 7f8d777fe640 1 -- 172.21.15.153:0/1532094586 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8d6c0015f0 con 0x7f8d7814a700 2024-03-27T03:10:44.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.520+0000 7f8d777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:44.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.520+0000 7f8d777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:10:44.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.520+0000 7f8d777fe640 10 monclient: dump: 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.521+0000 7f8d7f08d640 10 monclient: shutdown 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.521+0000 7f8d7f08d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.521+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1532094586 >> v1:172.21.15.196:6789/0 conn(0x7f8d7814a700 legacy=0x7f8d780abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.521+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1532094586 shutdown_connections 2024-03-27T03:10:44.517 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.521+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1532094586 >> 172.21.15.153:0/1532094586 conn(0x7f8d7806b660 msgr2=0x7f8d7806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:44.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.521+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1532094586 shutdown_connections 2024-03-27T03:10:44.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.521+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1532094586 wait complete. 2024-03-27T03:10:44.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.522+0000 7f8d7f08d640 10 monclient: build_initial_monmap 2024-03-27T03:10:44.518 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.522+0000 7f8d7f08d640 10 monclient: monmap: 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.522+0000 7f8d7f08d640 1 librados: starting msgr at 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.522+0000 7f8d7f08d640 1 librados: starting objecter 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.523+0000 7f8d7f08d640 1 Processor -- start 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.523+0000 7f8d7f08d640 1 -- start start 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.523+0000 7f8d7f08d640 1 librados: setting wanted keys 2024-03-27T03:10:44.519 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.523+0000 7f8d7f08d640 1 librados: calling monclient init 2024-03-27T03:10:44.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.523+0000 7f8d7f08d640 10 monclient: init 2024-03-27T03:10:44.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.524+0000 7f8d7f08d640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:44.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.524+0000 7f8d7f08d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:10:44.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.524+0000 7f8d7f08d640 10 monclient(hunting): picked mon.noname-c con 0x7f8d780ac5a0 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:44.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.526+0000 7f8d7f08d640 10 monclient(hunting): picked mon.noname-b con 0x7f8d780b0770 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:44.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.526+0000 7f8d7f08d640 10 monclient(hunting): picked mon.noname-a con 0x7f8d7814a700 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:44.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.526+0000 7f8d7f08d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d78123ee0 con 0x7f8d7814a700 2024-03-27T03:10:44.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.526+0000 7f8d7f08d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d78124130 con 0x7f8d780b0770 2024-03-27T03:10:44.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.526+0000 7f8d7f08d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d781d2fe0 con 0x7f8d780ac5a0 2024-03-27T03:10:44.520 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.526+0000 7f8d7f08d640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:44.521 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.526+0000 7f8d7f08d640 10 monclient(hunting): authenticate will time out at 1604.833149s 2024-03-27T03:10:44.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.529+0000 7f8d7ce29640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f8d780ac5a0 0x7f8d781cf820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:55039/0 (socket says 172.21.15.153:55039) 2024-03-27T03:10:44.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.529+0000 7f8d7ce29640 1 -- 172.21.15.153:0/1667704459 learned_addr learned my addr 172.21.15.153:0/1667704459 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:44.523 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.529+0000 7f8d757fa640 10 client.?.objecter ms_handle_connect 0x7f8d780ac5a0 2024-03-27T03:10:44.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.529+0000 7f8d757fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:44.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.532+0000 7f8d757fa640 10 client.?.objecter ms_handle_connect 0x7f8d7814a700 2024-03-27T03:10:44.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.532+0000 7f8d757fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:44.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.532+0000 7f8d757fa640 10 client.?.objecter ms_handle_connect 0x7f8d780b0770 2024-03-27T03:10:44.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.532+0000 7f8d757fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:44.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.532+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791553339 0 0) 0x7f8d78124130 con 0x7f8d780b0770 2024-03-27T03:10:44.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.532+0000 7f8d757fa640 10 monclient(hunting): _negotiate 2024-03-27T03:10:44.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.532+0000 7f8d757fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:44.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.533+0000 7f8d757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:44.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.533+0000 7f8d757fa640 10 monclient(hunting): my global_id is 4426 2024-03-27T03:10:44.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.533+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d64004eb0 con 0x7f8d780b0770 2024-03-27T03:10:44.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.533+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841488066 0 0) 0x7f8d781d2fe0 con 0x7f8d780ac5a0 2024-03-27T03:10:44.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.533+0000 7f8d757fa640 10 monclient(hunting): _negotiate 2024-03-27T03:10:44.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.533+0000 7f8d757fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:44.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.533+0000 7f8d757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:44.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.533+0000 7f8d757fa640 10 monclient(hunting): my global_id is 4445 2024-03-27T03:10:44.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.533+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d78124130 con 0x7f8d780ac5a0 2024-03-27T03:10:44.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.533+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3327837169 0 0) 0x7f8d78124130 con 0x7f8d780ac5a0 2024-03-27T03:10:44.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.533+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d64005490 con 0x7f8d780ac5a0 2024-03-27T03:10:44.528 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705102343 0 0) 0x7f8d78123ee0 con 0x7f8d7814a700 2024-03-27T03:10:44.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 10 monclient(hunting): _negotiate 2024-03-27T03:10:44.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:44.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:44.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 10 monclient(hunting): my global_id is 4404 2024-03-27T03:10:44.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d64005920 con 0x7f8d7814a700 2024-03-27T03:10:44.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3700438305 0 0) 0x7f8d64004eb0 con 0x7f8d780b0770 2024-03-27T03:10:44.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d78123ee0 con 0x7f8d780b0770 2024-03-27T03:10:44.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8d6c0015f0 con 0x7f8d780b0770 2024-03-27T03:10:44.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:44.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 10 monclient(hunting): dump: 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:10:44.530 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8d68001890 con 0x7f8d780ac5a0 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 10 monclient(hunting): dump: 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:44.531 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:44.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.534+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3230888485 0 0) 0x7f8d64005490 con 0x7f8d780ac5a0 2024-03-27T03:10:44.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 >> v1:172.21.15.196:6789/0 conn(0x7f8d780b0770 legacy=0x7f8d781df890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:44.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 >> v1:172.21.15.153:6789/0 conn(0x7f8d7814a700 legacy=0x7f8d781e6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:44.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 10 monclient: _finish_hunting 0 2024-03-27T03:10:44.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 1 monclient: found mon.c 2024-03-27T03:10:44.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:44.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:44.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d781d2c00 con 0x7f8d780ac5a0 2024-03-27T03:10:44.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 10 monclient: _finish_auth 0 2024-03-27T03:10:44.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 10 monclient: _check_auth_tickets 2024-03-27T03:10:44.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:44.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d680047e0 con 0x7f8d780ac5a0 2024-03-27T03:10:44.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:44.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d76ffd640 4 set_mon_vals no callback set 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d7f08d640 5 monclient: authenticate success, global_id 4445 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d7f08d640 10 monclient: _renew_subs 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d7f08d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d78144bf0 con 0x7f8d780ac5a0 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d7f08d640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d7f08d640 10 monclient: _renew_subs 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d7f08d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d78144f90 con 0x7f8d780ac5a0 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d7f08d640 1 librados: init done 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.535+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8d68001890 con 0x7f8d780ac5a0 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.536+0000 7f8d757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.536+0000 7f8d757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.536+0000 7f8d757fa640 10 monclient: dump: 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:44.535 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:44.536 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:44.536 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:44.536 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:44.536 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:44.536 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:44.536 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:44.536 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:44.536 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:44.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.537+0000 7f8d7f08d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:44.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.537+0000 7f8d7f08d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:44.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.537+0000 7f8d7f08d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:44.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.537+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d781d3170 con 0x7f8d780ac5a0 2024-03-27T03:10:44.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.541+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f8d6801a7c0 con 0x7f8d780ac5a0 2024-03-27T03:10:44.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.542+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f8d68084710 con 0x7f8d780ac5a0 2024-03-27T03:10:44.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.542+0000 7f8d757fa640 10 client.4445.objecter ms_dispatch 0x7f8d78000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:10:44.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.542+0000 7f8d757fa640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:10:44.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.542+0000 7f8d757fa640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:10:44.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.542+0000 7f8d757fa640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-27T03:10:44.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.542+0000 7f8d77fff640 10 monclient: get_auth_request con 0x7f8d6406e610 auth_method 0 2024-03-27T03:10:44.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.543+0000 7f8d757fa640 10 client.4445.objecter ms_handle_connect 0x7f8d6406e610 2024-03-27T03:10:44.544 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.550+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d6804fe20 con 0x7f8d780ac5a0 2024-03-27T03:10:44.544 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.551+0000 7f8d757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:44.545 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.551+0000 7f8d757fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:10:44.862 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.868+0000 7f8d7f08d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:44.862 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.868+0000 7f8d7f08d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:44.862 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.868+0000 7f8d7f08d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:10:44.862 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.868+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d781d3170 con 0x7f8d780ac5a0 2024-03-27T03:10:44.866 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d7ce29640 1 -- 172.21.15.153:0/1667704459 >> v1:172.21.15.153:6790/0 conn(0x7f8d780ac5a0 legacy=0x7f8d781cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-27T03:10:44.866 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d7ce29640 1 -- 172.21.15.153:0/1667704459 >> v1:172.21.15.153:6790/0 conn(0x7f8d780ac5a0 legacy=0x7f8d781cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-27T03:10:44.866 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d7ce29640 1 --1- 172.21.15.153:0/1667704459 >> v1:172.21.15.153:6790/0 conn(0x7f8d780ac5a0 0x7f8d781cf820 :-1 s=OPENED pgs=473370 cs=1 l=1).handle_message read tag failed 2024-03-27T03:10:44.866 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d7ce29640 1 --1- 172.21.15.153:0/1667704459 >> v1:172.21.15.153:6790/0 conn(0x7f8d780ac5a0 0x7f8d781cf820 :-1 s=OPENED pgs=473370 cs=1 l=1).fault on lossy channel, failing 2024-03-27T03:10:44.866 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d757fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.153:6790/0 2024-03-27T03:10:44.866 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d757fa640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:44.867 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 >> v1:172.21.15.153:6790/0 conn(0x7f8d780ac5a0 legacy=0x7f8d781cf820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T03:10:44.868 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d757fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d757fa640 10 monclient(hunting): picked mon.a con 0x7f8d7814a700 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d757fa640 10 monclient(hunting): picked mon.b con 0x7f8d780b0770 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d757fa640 10 monclient(hunting): picked mon.c con 0x7f8d6407aba0 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8d64004eb0 con 0x7f8d7814a700 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8d6407fcb0 con 0x7f8d780b0770 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8d64080e70 con 0x7f8d6407aba0 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.871+0000 7f8d757fa640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.875+0000 7f8d757fa640 10 client.4445.objecter ms_handle_connect 0x7f8d780b0770 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.875+0000 7f8d757fa640 10 client.4445.objecter resend_mon_ops 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.875+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724605046 0 0) 0x7f8d6407fcb0 con 0x7f8d780b0770 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.875+0000 7f8d757fa640 10 monclient(hunting): _negotiate 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.875+0000 7f8d757fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:44.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.875+0000 7f8d757fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T03:10:44.871 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.875+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.196:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f8d6407fe40 con 0x7f8d780b0770 2024-03-27T03:10:44.871 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.875+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1908352534 0 0) 0x7f8d6407fe40 con 0x7f8d780b0770 2024-03-27T03:10:44.871 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.875+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 >> v1:172.21.15.153:6790/0 conn(0x7f8d6407aba0 legacy=0x7f8d6407d030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:10:44.871 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.875+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 >> v1:172.21.15.153:6789/0 conn(0x7f8d7814a700 legacy=0x7f8d64075680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.877+0000 7f8d757fa640 10 monclient: _finish_hunting 0 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.877+0000 7f8d757fa640 1 monclient: found mon.b 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.877+0000 7f8d757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.877+0000 7f8d757fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.877+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f8d64082080 con 0x7f8d780b0770 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.878+0000 7f8d757fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.878+0000 7f8d757fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.878+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d64004eb0 con 0x7f8d780b0770 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.878+0000 7f8d757fa640 10 monclient: _finish_auth 0 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.878+0000 7f8d757fa640 10 monclient: _check_auth_tickets 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.878+0000 7f8d757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.878+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f8d70001a80 con 0x7f8d780b0770 2024-03-27T03:10:44.873 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.878+0000 7f8d757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:44.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.878+0000 7f8d757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:10:44.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.878+0000 7f8d757fa640 10 monclient: dump: 2024-03-27T03:10:44.874 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:44.874 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:44.874 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:44.874 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:44.874 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:44.874 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:44.874 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:44.874 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:44.874 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:44.874 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:44.878 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:10:44.878 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:10:44.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.883+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d700018a0 con 0x7f8d780b0770 2024-03-27T03:10:44.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.883+0000 7f8d757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:44.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.883+0000 7f8d76ffd640 4 set_mon_vals no callback set 2024-03-27T03:10:44.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.883+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f8d70001ca0 con 0x7f8d780b0770 2024-03-27T03:10:44.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.884+0000 7f8d757fa640 1 -- 172.21.15.153:0/1667704459 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f8d70084be0 con 0x7f8d780b0770 2024-03-27T03:10:44.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.884+0000 7f8d757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:44.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.884+0000 7f8d757fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:10:44.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.888+0000 7f8d7f08d640 10 librados: watch_flush enter 2024-03-27T03:10:44.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.888+0000 7f8d7f08d640 10 librados: watch_flush exit 2024-03-27T03:10:44.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.888+0000 7f8d7f08d640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-27T03:10:44.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.888+0000 7f8d7f08d640 10 client.4445.objecter successfully canceled tick 2024-03-27T03:10:44.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.888+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1667704459 >> v1:172.21.15.153:6816/2637937781 conn(0x7f8d6406e610 legacy=0x7f8d64070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:44.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.888+0000 7f8d7f08d640 10 monclient: shutdown 2024-03-27T03:10:44.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.888+0000 7f8d7f08d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:44.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.888+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1667704459 >> v1:172.21.15.196:6789/0 conn(0x7f8d780b0770 legacy=0x7f8d64079030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:44.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.888+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1667704459 shutdown_connections 2024-03-27T03:10:44.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.888+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1667704459 >> 172.21.15.153:0/1667704459 conn(0x7f8d7806b660 msgr2=0x7f8d780b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:44.884 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.889+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1667704459 shutdown_connections 2024-03-27T03:10:44.884 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.889+0000 7f8d7f08d640 1 -- 172.21.15.153:0/1667704459 wait complete. 2024-03-27T03:10:44.884 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:44.889+0000 7f8d7f08d640 1 librados: shutdown 2024-03-27T03:10:44.906 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:10:51.713 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:10:51.868 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.875+0000 7fe926115640 10 monclient: get_monmap_and_config 2024-03-27T03:10:51.869 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.875+0000 7fe926115640 10 monclient: build_initial_monmap 2024-03-27T03:10:51.869 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.875+0000 7fe926115640 10 monclient: monmap: 2024-03-27T03:10:51.869 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:51.869 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:51.869 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:10:51.876623+0000 2024-03-27T03:10:51.869 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:10:51.876623+0000 2024-03-27T03:10:51.869 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:51.869 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:51.869 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:51.869 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:10:51.869 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:10:51.869 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:51.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.877+0000 7fe926115640 1 Processor -- start 2024-03-27T03:10:51.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.877+0000 7fe926115640 1 -- start start 2024-03-27T03:10:51.870 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.877+0000 7fe926115640 10 monclient: init 2024-03-27T03:10:51.871 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.878+0000 7fe926115640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:51.871 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.878+0000 7fe926115640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:10:51.871 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.878+0000 7fe926115640 10 monclient(hunting): picked mon.noname-c con 0x7fe9201457f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:51.872 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.878+0000 7fe926115640 10 monclient(hunting): picked mon.noname-a con 0x7fe92013f650 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:51.872 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.878+0000 7fe926115640 10 monclient(hunting): picked mon.noname-b con 0x7fe9200a9d70 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:51.872 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.878+0000 7fe926115640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9200aaef0 con 0x7fe92013f650 2024-03-27T03:10:51.872 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.878+0000 7fe926115640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe920145bd0 con 0x7fe9201457f0 2024-03-27T03:10:51.872 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.878+0000 7fe926115640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe920145eb0 con 0x7fe9200a9d70 2024-03-27T03:10:51.872 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.878+0000 7fe926115640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:51.872 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.878+0000 7fe926115640 10 monclient(hunting): authenticate will time out at 1612.184333s 2024-03-27T03:10:51.872 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.878+0000 7fe91effd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe92013f650 0x7fe9200b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56742/0 (socket says 172.21.15.153:56742) 2024-03-27T03:10:51.872 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.878+0000 7fe91effd640 1 -- 172.21.15.153:0/3782709775 learned_addr learned my addr 172.21.15.153:0/3782709775 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:51.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.879+0000 7fe91e7fc640 1 -- 172.21.15.153:0/3782709775 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148053719 0 0) 0x7fe920145bd0 con 0x7fe9201457f0 2024-03-27T03:10:51.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.879+0000 7fe91e7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:10:51.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.879+0000 7fe91e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:51.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.879+0000 7fe91e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:51.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.879+0000 7fe91e7fc640 10 monclient(hunting): my global_id is 4429 2024-03-27T03:10:51.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.879+0000 7fe91e7fc640 1 -- 172.21.15.153:0/3782709775 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe908003520 con 0x7fe9201457f0 2024-03-27T03:10:51.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.879+0000 7fe91effd640 1 -- 172.21.15.153:0/3782709775 >> v1:172.21.15.153:6789/0 conn(0x7fe92013f650 legacy=0x7fe9200b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-27T03:10:51.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.879+0000 7fe91effd640 1 -- 172.21.15.153:0/3782709775 >> v1:172.21.15.153:6789/0 conn(0x7fe92013f650 legacy=0x7fe9200b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-27T03:10:51.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.879+0000 7fe91effd640 1 --1- 172.21.15.153:0/3782709775 >> v1:172.21.15.153:6789/0 conn(0x7fe92013f650 0x7fe9200b2620 :-1 s=OPENED pgs=505530 cs=1 l=1).handle_message read tag failed 2024-03-27T03:10:51.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.879+0000 7fe91effd640 1 --1- 172.21.15.153:0/3782709775 >> v1:172.21.15.153:6789/0 conn(0x7fe92013f650 0x7fe9200b2620 :-1 s=OPENED pgs=505530 cs=1 l=1).fault on lossy channel, failing 2024-03-27T03:10:51.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.879+0000 7fe91e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.153:6789/0 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 1 -- 172.21.15.153:0/3782709775 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4069486711 0 0) 0x7fe908003520 con 0x7fe9201457f0 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 1 -- 172.21.15.153:0/3782709775 >> v1:172.21.15.153:6790/0 conn(0x7fe9200a9d70 legacy=0x7fe92014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 1 -- 172.21.15.153:0/3782709775 >> v1:172.21.15.153:6789/0 conn(0x7fe92013f650 legacy=0x7fe9200b2620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 1 monclient: found mon.noname-c 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 1 -- 172.21.15.153:0/3782709775 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe920146340 con 0x7fe9201457f0 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 10 monclient: _finish_auth 0 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 1 -- 172.21.15.153:0/3782709775 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe918001a10 con 0x7fe9201457f0 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 10 monclient: dump: 2024-03-27T03:10:51.875 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe926115640 5 monclient: authenticate success, global_id 4429 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe926115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 1 -- 172.21.15.153:0/3782709775 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe918001b70 con 0x7fe9201457f0 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 1 -- 172.21.15.153:0/3782709775 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe918001f80 con 0x7fe9201457f0 2024-03-27T03:10:51.876 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:51.877 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:10:51.877 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.880+0000 7fe91e7fc640 10 monclient: dump: 2024-03-27T03:10:51.877 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:51.877 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:51.877 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:51.877 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:51.877 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:51.877 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:51.877 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:51.877 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:51.877 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:51.877 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:51.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.881+0000 7fe926115640 10 monclient: get_monmap_and_config success 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.881+0000 7fe926115640 4 set_mon_vals no callback set 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.882+0000 7fe926115640 10 monclient: shutdown 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.882+0000 7fe926115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.882+0000 7fe926115640 1 -- 172.21.15.153:0/3782709775 >> v1:172.21.15.196:6789/0 conn(0x7fe9201457f0 legacy=0x7fe92013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.882+0000 7fe926115640 1 -- 172.21.15.153:0/3782709775 shutdown_connections 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.882+0000 7fe926115640 1 -- 172.21.15.153:0/3782709775 >> 172.21.15.153:0/3782709775 conn(0x7fe920066a00 msgr2=0x7fe920066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.882+0000 7fe926115640 1 -- 172.21.15.153:0/3782709775 shutdown_connections 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.882+0000 7fe926115640 1 -- 172.21.15.153:0/3782709775 wait complete. 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.883+0000 7fe926115640 10 monclient: build_initial_monmap 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.883+0000 7fe926115640 10 monclient: monmap: 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:10:51.886 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.884+0000 7fe926115640 1 librados: starting msgr at 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.884+0000 7fe926115640 1 librados: starting objecter 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.884+0000 7fe926115640 1 Processor -- start 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.884+0000 7fe926115640 1 -- start start 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.884+0000 7fe926115640 1 librados: setting wanted keys 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.884+0000 7fe926115640 1 librados: calling monclient init 2024-03-27T03:10:51.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.884+0000 7fe926115640 10 monclient: init 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.885+0000 7fe926115640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.885+0000 7fe926115640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.885+0000 7fe926115640 10 monclient(hunting): picked mon.noname-c con 0x7fe9200a9d70 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe926115640 10 monclient(hunting): picked mon.noname-b con 0x7fe92013f650 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe91f7fe640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fe9200a9d70 0x7fe9201cd520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:42476/0 (socket says 172.21.15.153:42476) 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe91f7fe640 1 -- 172.21.15.153:0/3073233451 learned_addr learned my addr 172.21.15.153:0/3073233451 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe926115640 10 monclient(hunting): picked mon.noname-a con 0x7fe9201457f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9201ccbb0 con 0x7fe9201457f0 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe907fff640 10 client.?.objecter ms_handle_connect 0x7fe92013f650 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe907fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9201387a0 con 0x7fe92013f650 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe907fff640 10 client.?.objecter ms_handle_connect 0x7fe9200a9d70 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe907fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9201cce20 con 0x7fe9200a9d70 2024-03-27T03:10:51.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe926115640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe926115640 10 monclient(hunting): authenticate will time out at 1612.192244s 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe907fff640 10 client.?.objecter ms_handle_connect 0x7fe9201457f0 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.886+0000 7fe907fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296604179 0 0) 0x7fe9201387a0 con 0x7fe92013f650 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 10 monclient(hunting): _negotiate 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 10 monclient(hunting): my global_id is 4432 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe90c004eb0 con 0x7fe92013f650 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371708673 0 0) 0x7fe9201ccbb0 con 0x7fe9201457f0 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 10 monclient(hunting): _negotiate 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 10 monclient(hunting): my global_id is 4407 2024-03-27T03:10:51.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9201387a0 con 0x7fe9201457f0 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348941271 0 0) 0x7fe9201cce20 con 0x7fe9200a9d70 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 10 monclient(hunting): _negotiate 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 10 monclient(hunting): my global_id is 4451 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.887+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9201ccbb0 con 0x7fe9200a9d70 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 27179999 0 0) 0x7fe90c004eb0 con 0x7fe92013f650 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9201cce20 con 0x7fe92013f650 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe914001930 con 0x7fe92013f650 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 10 monclient(hunting): dump: 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:51.890 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3262638395 0 0) 0x7fe9201ccbb0 con 0x7fe9200a9d70 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe90c004eb0 con 0x7fe9200a9d70 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe918001e80 con 0x7fe9200a9d70 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 10 monclient(hunting): dump: 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:51.891 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1114143141 0 0) 0x7fe9201387a0 con 0x7fe9201457f0 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9201ccbb0 con 0x7fe9201457f0 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe910001830 con 0x7fe9201457f0 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 10 monclient(hunting): dump: 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:51.892 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.888+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 234967607 0 0) 0x7fe9201cce20 con 0x7fe92013f650 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 >> v1:172.21.15.153:6790/0 conn(0x7fe9200a9d70 legacy=0x7fe9201cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 >> v1:172.21.15.153:6789/0 conn(0x7fe9201457f0 legacy=0x7fe920138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 10 monclient: _finish_hunting 0 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 1 monclient: found mon.b 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9201cd100 con 0x7fe92013f650 2024-03-27T03:10:51.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 10 monclient: _finish_auth 0 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 10 monclient: _check_auth_tickets 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe926115640 5 monclient: authenticate success, global_id 4432 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe926115640 10 monclient: _renew_subs 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe926115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9201cbe80 con 0x7fe92013f650 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe926115640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe926115640 10 monclient: _renew_subs 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe926115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9201cde10 con 0x7fe92013f650 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe926115640 1 librados: init done 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe914001d60 con 0x7fe92013f650 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe914004a00 con 0x7fe92013f650 2024-03-27T03:10:51.894 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe91d7fa640 4 set_mon_vals no callback set 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.889+0000 7fe907fff640 10 monclient: dump: 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.891+0000 7fe926115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.891+0000 7fe926115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:51.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.891+0000 7fe926115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:10:51.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.891+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe900005080 con 0x7fe92013f650 2024-03-27T03:10:51.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.891+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fe914004c60 con 0x7fe92013f650 2024-03-27T03:10:51.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.892+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fe914084a40 con 0x7fe92013f650 2024-03-27T03:10:51.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.892+0000 7fe91f7fe640 10 monclient: get_auth_request con 0x7fe90c06e0a0 auth_method 0 2024-03-27T03:10:51.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.893+0000 7fe907fff640 10 client.4432.objecter ms_dispatch 0x7fe920000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:10:51.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.893+0000 7fe907fff640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:10:51.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.893+0000 7fe907fff640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:10:51.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.893+0000 7fe907fff640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-27T03:10:51.896 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.893+0000 7fe907fff640 10 client.4432.objecter ms_handle_connect 0x7fe90c06e0a0 2024-03-27T03:10:51.897 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.898+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe914050380 con 0x7fe92013f650 2024-03-27T03:10:51.897 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.898+0000 7fe907fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:51.897 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:51.898+0000 7fe907fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:10:52.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.278+0000 7fe926115640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:52.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.278+0000 7fe926115640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:52.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.278+0000 7fe926115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:10:52.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.278+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe900005330 con 0x7fe92013f650 2024-03-27T03:10:52.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.279+0000 7fe907fff640 1 -- 172.21.15.153:0/3073233451 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fe9140840f0 con 0x7fe92013f650 2024-03-27T03:10:52.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.279+0000 7fe907fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:52.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.279+0000 7fe907fff640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:10:52.273 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:10:52.273 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:10:52.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.283+0000 7fe926115640 10 librados: watch_flush enter 2024-03-27T03:10:52.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.283+0000 7fe926115640 10 librados: watch_flush exit 2024-03-27T03:10:52.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.283+0000 7fe926115640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-27T03:10:52.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.283+0000 7fe926115640 10 client.4432.objecter successfully canceled tick 2024-03-27T03:10:52.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.283+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 >> v1:172.21.15.153:6816/2637937781 conn(0x7fe90c06e0a0 legacy=0x7fe90c070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:52.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.283+0000 7fe926115640 10 monclient: shutdown 2024-03-27T03:10:52.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.283+0000 7fe926115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:52.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.283+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 >> v1:172.21.15.196:6789/0 conn(0x7fe92013f650 legacy=0x7fe920137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:52.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.283+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 shutdown_connections 2024-03-27T03:10:52.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.284+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 >> 172.21.15.153:0/3073233451 conn(0x7fe920066a00 msgr2=0x7fe9200b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:52.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.284+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 shutdown_connections 2024-03-27T03:10:52.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.284+0000 7fe926115640 1 -- 172.21.15.153:0/3073233451 wait complete. 2024-03-27T03:10:52.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:52.284+0000 7fe926115640 1 librados: shutdown 2024-03-27T03:10:52.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:10:59.098 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:10:59.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.260+0000 7fc28b59e640 10 monclient: get_monmap_and_config 2024-03-27T03:10:59.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.260+0000 7fc28b59e640 10 monclient: build_initial_monmap 2024-03-27T03:10:59.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.260+0000 7fc28b59e640 10 monclient: monmap: 2024-03-27T03:10:59.254 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:59.254 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:59.254 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:10:59.260996+0000 2024-03-27T03:10:59.254 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:10:59.260996+0000 2024-03-27T03:10:59.254 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:59.254 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:59.254 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:59.254 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:10:59.255 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:10:59.255 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:59.255 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.261+0000 7fc28b59e640 1 Processor -- start 2024-03-27T03:10:59.255 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.261+0000 7fc28b59e640 1 -- start start 2024-03-27T03:10:59.255 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.261+0000 7fc28b59e640 10 monclient: init 2024-03-27T03:10:59.255 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.262+0000 7fc28b59e640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:59.255 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.262+0000 7fc28b59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:10:59.255 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.262+0000 7fc28b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc28c0ae220 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:59.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.262+0000 7fc28b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc28c072f10 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:59.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.262+0000 7fc28b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc28c0ad730 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:59.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.262+0000 7fc28b59e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0aa9f0 con 0x7fc28c0ae220 2024-03-27T03:10:59.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.262+0000 7fc28b59e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0ac4c0 con 0x7fc28c072f10 2024-03-27T03:10:59.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.262+0000 7fc28b59e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0ac7a0 con 0x7fc28c0ad730 2024-03-27T03:10:59.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.262+0000 7fc28b59e640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:59.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.262+0000 7fc28b59e640 10 monclient(hunting): authenticate will time out at 1619.567974s 2024-03-27T03:10:59.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.263+0000 7fc28a59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc28c0ae220 0x7fc28c141450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36956/0 (socket says 172.21.15.153:36956) 2024-03-27T03:10:59.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.263+0000 7fc28a59c640 1 -- 172.21.15.153:0/3662359545 learned_addr learned my addr 172.21.15.153:0/3662359545 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:59.260 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.266+0000 7fc28959a640 1 -- 172.21.15.153:0/3662359545 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222271353 0 0) 0x7fc28c0ac7a0 con 0x7fc28c0ad730 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.266+0000 7fc28959a640 10 monclient(hunting): _negotiate 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.266+0000 7fc28959a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.266+0000 7fc28959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.266+0000 7fc28959a640 10 monclient(hunting): my global_id is 4454 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.266+0000 7fc28959a640 1 -- 172.21.15.153:0/3662359545 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc274003520 con 0x7fc28c0ad730 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 1 -- 172.21.15.153:0/3662359545 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3103498460 0 0) 0x7fc274003520 con 0x7fc28c0ad730 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 1 -- 172.21.15.153:0/3662359545 >> v1:172.21.15.196:6789/0 conn(0x7fc28c072f10 legacy=0x7fc28c141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 1 -- 172.21.15.153:0/3662359545 >> v1:172.21.15.153:6789/0 conn(0x7fc28c0ae220 legacy=0x7fc28c141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 10 monclient: _finish_hunting 0 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 1 monclient: found mon.noname-b 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 1 -- 172.21.15.153:0/3662359545 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c0adb40 con 0x7fc28c0ad730 2024-03-27T03:10:59.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 10 monclient: _finish_auth 0 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 10 monclient: _check_auth_tickets 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 1 -- 172.21.15.153:0/3662359545 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc280001670 con 0x7fc28c0ad730 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 10 monclient: dump: 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:59.262 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28b59e640 5 monclient: authenticate success, global_id 4454 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 1 -- 172.21.15.153:0/3662359545 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc280001900 con 0x7fc28c0ad730 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28b59e640 10 monclient: get_monmap_and_config success 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28b59e640 4 set_mon_vals no callback set 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 1 -- 172.21.15.153:0/3662359545 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc280001670 con 0x7fc28c0ad730 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.267+0000 7fc28959a640 10 monclient: dump: 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:59.263 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:59.264 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:59.264 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:59.264 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:59.264 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:59.264 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:59.264 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:59.264 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:59.265 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.268+0000 7fc28b59e640 10 monclient: shutdown 2024-03-27T03:10:59.265 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.268+0000 7fc28b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.268+0000 7fc28b59e640 1 -- 172.21.15.153:0/3662359545 >> v1:172.21.15.153:6790/0 conn(0x7fc28c0ad730 legacy=0x7fc28c1497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.270+0000 7fc28b59e640 1 -- 172.21.15.153:0/3662359545 shutdown_connections 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.270+0000 7fc28b59e640 1 -- 172.21.15.153:0/3662359545 >> 172.21.15.153:0/3662359545 conn(0x7fc28c0b3a30 msgr2=0x7fc28c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.270+0000 7fc28b59e640 1 -- 172.21.15.153:0/3662359545 shutdown_connections 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.270+0000 7fc28b59e640 1 -- 172.21.15.153:0/3662359545 wait complete. 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.271+0000 7fc28b59e640 10 monclient: build_initial_monmap 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.271+0000 7fc28b59e640 10 monclient: monmap: 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:10:59.266 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:10:59.267 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:10:59.267 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:59.267 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.271+0000 7fc28b59e640 1 librados: starting msgr at 2024-03-27T03:10:59.267 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.271+0000 7fc28b59e640 1 librados: starting objecter 2024-03-27T03:10:59.267 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.272+0000 7fc28b59e640 1 Processor -- start 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.272+0000 7fc28b59e640 1 -- start start 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.272+0000 7fc28b59e640 1 librados: setting wanted keys 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.272+0000 7fc28b59e640 1 librados: calling monclient init 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.272+0000 7fc28b59e640 10 monclient: init 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.273+0000 7fc28b59e640 10 monclient: _reopen_session rank -1 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.273+0000 7fc28b59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.273+0000 7fc28b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc28c072f10 addr v1:172.21.15.196:6789/0 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.273+0000 7fc28b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc28c0ad730 addr v1:172.21.15.153:6790/0 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.273+0000 7fc28b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc28c0ae220 addr v1:172.21.15.153:6789/0 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.273+0000 7fc28b59e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c06e5a0 con 0x7fc28c0ae220 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.273+0000 7fc28b59e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c070400 con 0x7fc28c072f10 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.273+0000 7fc28b59e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0706e0 con 0x7fc28c0ad730 2024-03-27T03:10:59.270 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.273+0000 7fc28b59e640 10 monclient(hunting): _renew_subs 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.273+0000 7fc28b59e640 10 monclient(hunting): authenticate will time out at 1619.579005s 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.274+0000 7fc28ad9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc28c0ae220 0x7fc28c1e63c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:38624/0 (socket says 172.21.15.153:38624) 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.274+0000 7fc28ad9d640 1 -- 172.21.15.153:0/1989678878 learned_addr learned my addr 172.21.15.153:0/1989678878 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.274+0000 7fc27affd640 10 client.?.objecter ms_handle_connect 0x7fc28c0ae220 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.274+0000 7fc27affd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.274+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049885019 0 0) 0x7fc28c06e5a0 con 0x7fc28c0ae220 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.274+0000 7fc27affd640 10 monclient(hunting): _negotiate 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.275+0000 7fc27affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.275+0000 7fc27affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.275+0000 7fc27affd640 10 monclient(hunting): my global_id is 4410 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.275+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc26c004eb0 con 0x7fc28c0ae220 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.275+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 784826281 0 0) 0x7fc26c004eb0 con 0x7fc28c0ae220 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.275+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc28c06e5a0 con 0x7fc28c0ae220 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.275+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc280001670 con 0x7fc28c0ae220 2024-03-27T03:10:59.271 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.276+0000 7fc27affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.276+0000 7fc27affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.276+0000 7fc27affd640 10 monclient(hunting): dump: 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.276+0000 7fc27affd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.276+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1450433291 0 0) 0x7fc28c06e5a0 con 0x7fc28c0ae220 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.276+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 >> v1:172.21.15.153:6790/0 conn(0x7fc28c0ad730 legacy=0x7fc28c1dee80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:10:59.272 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.276+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 >> v1:172.21.15.196:6789/0 conn(0x7fc28c072f10 legacy=0x7fc28c06ea40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:10:59.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.280+0000 7fc27affd640 10 monclient: _finish_hunting 0 2024-03-27T03:10:59.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.280+0000 7fc27affd640 1 monclient: found mon.a 2024-03-27T03:10:59.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.280+0000 7fc27affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:10:59.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.280+0000 7fc27affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:10:59.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.280+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c06e080 con 0x7fc28c0ae220 2024-03-27T03:10:59.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.280+0000 7fc27affd640 10 monclient: _finish_auth 0 2024-03-27T03:10:59.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.280+0000 7fc27affd640 10 monclient: _check_auth_tickets 2024-03-27T03:10:59.274 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.281+0000 7fc27affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:10:59.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.281+0000 7fc28b59e640 5 monclient: authenticate success, global_id 4410 2024-03-27T03:10:59.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.281+0000 7fc28b59e640 10 monclient: _renew_subs 2024-03-27T03:10:59.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.281+0000 7fc28b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:10:59.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.281+0000 7fc28b59e640 1 -- 172.21.15.153:0/1989678878 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c06f150 con 0x7fc28c0ae220 2024-03-27T03:10:59.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.281+0000 7fc28b59e640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:10:59.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.281+0000 7fc28b59e640 10 monclient: _renew_subs 2024-03-27T03:10:59.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.281+0000 7fc28b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:10:59.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.281+0000 7fc28b59e640 1 -- 172.21.15.153:0/1989678878 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c1df770 con 0x7fc28c0ae220 2024-03-27T03:10:59.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.281+0000 7fc28b59e640 1 librados: init done 2024-03-27T03:10:59.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.282+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc280001c50 con 0x7fc28c0ae220 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.282+0000 7fc27affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.282+0000 7fc28b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.282+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc280003ee0 con 0x7fc28c0ae220 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.282+0000 7fc28b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.282+0000 7fc28b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.282+0000 7fc288d99640 4 set_mon_vals no callback set 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.282+0000 7fc28b59e640 1 -- 172.21.15.153:0/1989678878 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c04f730 con 0x7fc28c0ae220 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.282+0000 7fc27affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.282+0000 7fc27affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.282+0000 7fc27affd640 10 monclient: dump: 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:59.276 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:10:59.277 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:10:59.277 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:10:59.277 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:10:59.277 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:10:59.277 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:10:59.277 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:10:59.279 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.286+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fc280004040 con 0x7fc28c0ae220 2024-03-27T03:10:59.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.287+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fc280084c20 con 0x7fc28c0ae220 2024-03-27T03:10:59.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.287+0000 7fc27affd640 10 client.4410.objecter ms_dispatch 0x7fc28c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:10:59.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.287+0000 7fc27affd640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:10:59.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.287+0000 7fc27affd640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:10:59.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.287+0000 7fc28a59c640 10 monclient: get_auth_request con 0x7fc26c06dcd0 auth_method 0 2024-03-27T03:10:59.281 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.287+0000 7fc27affd640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-27T03:10:59.281 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.287+0000 7fc27affd640 10 client.4410.objecter ms_handle_connect 0x7fc26c06dcd0 2024-03-27T03:10:59.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.294+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc280050500 con 0x7fc28c0ae220 2024-03-27T03:10:59.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.294+0000 7fc27affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:10:59.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.294+0000 7fc27affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:10:59.634 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.641+0000 7fc28b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:59.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.641+0000 7fc28b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:59.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.641+0000 7fc28b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:10:59.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.641+0000 7fc28b59e640 1 -- 172.21.15.153:0/1989678878 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc28c072b90 con 0x7fc28c0ae220 2024-03-27T03:10:59.641 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:10:59.642 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:10:59.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.645+0000 7fc27affd640 1 -- 172.21.15.153:0/1989678878 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fc28c072b90 con 0x7fc28c0ae220 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.645+0000 7fc27affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.645+0000 7fc27affd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.649+0000 7fc28b59e640 10 librados: watch_flush enter 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.649+0000 7fc28b59e640 10 librados: watch_flush exit 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.649+0000 7fc28b59e640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.649+0000 7fc28b59e640 10 client.4410.objecter successfully canceled tick 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.649+0000 7fc28b59e640 1 -- 172.21.15.153:0/1989678878 >> v1:172.21.15.153:6816/2637937781 conn(0x7fc26c06dcd0 legacy=0x7fc26c070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.649+0000 7fc28b59e640 10 monclient: shutdown 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.649+0000 7fc28b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.649+0000 7fc28b59e640 1 -- 172.21.15.153:0/1989678878 >> v1:172.21.15.153:6789/0 conn(0x7fc28c0ae220 legacy=0x7fc28c1e63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.649+0000 7fc28b59e640 1 -- 172.21.15.153:0/1989678878 shutdown_connections 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.649+0000 7fc28b59e640 1 -- 172.21.15.153:0/1989678878 >> 172.21.15.153:0/1989678878 conn(0x7fc28c0b3a30 msgr2=0x7fc28c0b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.649+0000 7fc28b59e640 1 -- 172.21.15.153:0/1989678878 shutdown_connections 2024-03-27T03:10:59.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.649+0000 7fc28b59e640 1 -- 172.21.15.153:0/1989678878 wait complete. 2024-03-27T03:10:59.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:10:59.650+0000 7fc28b59e640 1 librados: shutdown 2024-03-27T03:10:59.660 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:11:06.467 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:11:06.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.608+0000 7fc3ca684640 10 monclient: get_monmap_and_config 2024-03-27T03:11:06.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.608+0000 7fc3ca684640 10 monclient: build_initial_monmap 2024-03-27T03:11:06.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.608+0000 7fc3ca684640 10 monclient: monmap: 2024-03-27T03:11:06.602 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:06.602 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:06.602 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:11:06.609613+0000 2024-03-27T03:11:06.602 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:11:06.609613+0000 2024-03-27T03:11:06.602 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:06.602 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:06.602 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:06.602 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:11:06.602 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:11:06.603 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:06.603 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.609+0000 7fc3ca684640 1 Processor -- start 2024-03-27T03:11:06.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.609+0000 7fc3ca684640 1 -- start start 2024-03-27T03:11:06.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.609+0000 7fc3ca684640 10 monclient: init 2024-03-27T03:11:06.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.610+0000 7fc3ca684640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:06.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.610+0000 7fc3ca684640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:11:06.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.610+0000 7fc3ca684640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c41422b0 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:06.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.611+0000 7fc3c3fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc3c41422b0 0x7fc3c413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:33564/0 (socket says 172.21.15.153:33564) 2024-03-27T03:11:06.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.611+0000 7fc3ca684640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c4142690 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:06.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.611+0000 7fc3ca684640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c40b17d0 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:06.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.611+0000 7fc3ca684640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c4142c80 con 0x7fc3c4142690 2024-03-27T03:11:06.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.611+0000 7fc3ca684640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c40aae40 con 0x7fc3c41422b0 2024-03-27T03:11:06.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.611+0000 7fc3ca684640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c413fbd0 con 0x7fc3c40b17d0 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.611+0000 7fc3ca684640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.611+0000 7fc3ca684640 10 monclient(hunting): authenticate will time out at 1626.917419s 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.611+0000 7fc3c3fff640 1 -- 172.21.15.153:0/1630615430 learned_addr learned my addr 172.21.15.153:0/1630615430 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.612+0000 7fc3c2ffd640 1 -- 172.21.15.153:0/1630615430 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726125991 0 0) 0x7fc3c40aae40 con 0x7fc3c41422b0 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.612+0000 7fc3c2ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.612+0000 7fc3c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.612+0000 7fc3c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.612+0000 7fc3c2ffd640 10 monclient(hunting): my global_id is 4435 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.612+0000 7fc3c2ffd640 1 -- 172.21.15.153:0/1630615430 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac003520 con 0x7fc3c41422b0 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.612+0000 7fc3c2ffd640 1 -- 172.21.15.153:0/1630615430 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806332571 0 0) 0x7fc3c413fbd0 con 0x7fc3c40b17d0 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.612+0000 7fc3c2ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.612+0000 7fc3c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.612+0000 7fc3c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:06.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.612+0000 7fc3c2ffd640 10 monclient(hunting): my global_id is 4457 2024-03-27T03:11:06.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.612+0000 7fc3c2ffd640 1 -- 172.21.15.153:0/1630615430 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c40aae40 con 0x7fc3c40b17d0 2024-03-27T03:11:06.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 1 -- 172.21.15.153:0/1630615430 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1525506170 0 0) 0x7fc3c40aae40 con 0x7fc3c40b17d0 2024-03-27T03:11:06.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 1 -- 172.21.15.153:0/1630615430 >> v1:172.21.15.196:6789/0 conn(0x7fc3c41422b0 legacy=0x7fc3c413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:06.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 1 -- 172.21.15.153:0/1630615430 >> v1:172.21.15.153:6789/0 conn(0x7fc3c4142690 legacy=0x7fc3c40b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:11:06.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:11:06.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 1 monclient: found mon.noname-b 2024-03-27T03:11:06.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:06.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:11:06.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 1 -- 172.21.15.153:0/1630615430 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c4151940 con 0x7fc3c40b17d0 2024-03-27T03:11:06.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 10 monclient: _finish_auth 0 2024-03-27T03:11:06.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:11:06.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 1 -- 172.21.15.153:0/1630615430 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc3bc0019f0 con 0x7fc3c40b17d0 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3c2ffd640 10 monclient: dump: 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:06.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3ca684640 5 monclient: authenticate success, global_id 4457 2024-03-27T03:11:06.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.613+0000 7fc3ca684640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:11:06.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.614+0000 7fc3c2ffd640 1 -- 172.21.15.153:0/1630615430 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3bc001b50 con 0x7fc3c40b17d0 2024-03-27T03:11:06.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.614+0000 7fc3c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:06.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.614+0000 7fc3c2ffd640 1 -- 172.21.15.153:0/1630615430 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc3bc001f20 con 0x7fc3c40b17d0 2024-03-27T03:11:06.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.614+0000 7fc3c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:06.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.614+0000 7fc3c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:11:06.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.614+0000 7fc3c2ffd640 10 monclient: dump: 2024-03-27T03:11:06.611 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:06.611 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:06.611 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:06.611 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:06.611 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.614+0000 7fc3ca684640 10 monclient: get_monmap_and_config success 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.614+0000 7fc3ca684640 4 set_mon_vals no callback set 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.615+0000 7fc3ca684640 10 monclient: shutdown 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.615+0000 7fc3ca684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.615+0000 7fc3ca684640 1 -- 172.21.15.153:0/1630615430 >> v1:172.21.15.153:6790/0 conn(0x7fc3c40b17d0 legacy=0x7fc3c40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.615+0000 7fc3ca684640 1 -- 172.21.15.153:0/1630615430 shutdown_connections 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.615+0000 7fc3ca684640 1 -- 172.21.15.153:0/1630615430 >> 172.21.15.153:0/1630615430 conn(0x7fc3c4066a00 msgr2=0x7fc3c4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.615+0000 7fc3ca684640 1 -- 172.21.15.153:0/1630615430 shutdown_connections 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.615+0000 7fc3ca684640 1 -- 172.21.15.153:0/1630615430 wait complete. 2024-03-27T03:11:06.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.616+0000 7fc3ca684640 10 monclient: build_initial_monmap 2024-03-27T03:11:06.613 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.616+0000 7fc3ca684640 10 monclient: monmap: 2024-03-27T03:11:06.613 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:06.613 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:11:06.613 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:11:06.613 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:11:06.613 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:06.613 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:06.613 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:06.613 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:11:06.613 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:11:06.613 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:06.622 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.616+0000 7fc3ca684640 1 librados: starting msgr at 2024-03-27T03:11:06.622 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.616+0000 7fc3ca684640 1 librados: starting objecter 2024-03-27T03:11:06.622 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.616+0000 7fc3ca684640 1 Processor -- start 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.617+0000 7fc3ca684640 1 -- start start 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.617+0000 7fc3ca684640 1 librados: setting wanted keys 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.617+0000 7fc3ca684640 1 librados: calling monclient init 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.617+0000 7fc3ca684640 10 monclient: init 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.617+0000 7fc3ca684640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.617+0000 7fc3ca684640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.617+0000 7fc3ca684640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c40b17d0 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.617+0000 7fc3ca684640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c41422b0 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.617+0000 7fc3ca684640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c4142690 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.617+0000 7fc3c37fe640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fc3c41422b0 0x7fc3c412e0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:40156/0 (socket says 172.21.15.153:40156) 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.617+0000 7fc3c37fe640 1 -- 172.21.15.153:0/2206030102 learned_addr learned my addr 172.21.15.153:0/2206030102 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.617+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c40ac8f0 con 0x7fc3c40b17d0 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.618+0000 7fc3c0ff9640 10 client.?.objecter ms_handle_connect 0x7fc3c41422b0 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.618+0000 7fc3c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:06.623 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.618+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c4131c30 con 0x7fc3c4142690 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.618+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c40acb60 con 0x7fc3c41422b0 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.618+0000 7fc3ca684640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.618+0000 7fc3ca684640 10 monclient(hunting): authenticate will time out at 1626.923742s 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.618+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934313525 0 0) 0x7fc3c40acb60 con 0x7fc3c41422b0 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.618+0000 7fc3c0ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.618+0000 7fc3c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.618+0000 7fc3c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.618+0000 7fc3c0ff9640 10 monclient(hunting): my global_id is 4460 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.618+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3b0004eb0 con 0x7fc3c41422b0 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.619+0000 7fc3c0ff9640 10 client.?.objecter ms_handle_connect 0x7fc3c4142690 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.619+0000 7fc3c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.619+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651074179 0 0) 0x7fc3c4131c30 con 0x7fc3c4142690 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.619+0000 7fc3c0ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.619+0000 7fc3c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:06.624 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.619+0000 7fc3c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:06.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.619+0000 7fc3c0ff9640 10 monclient(hunting): my global_id is 4438 2024-03-27T03:11:06.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.619+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c40acb60 con 0x7fc3c4142690 2024-03-27T03:11:06.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.619+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1507347686 0 0) 0x7fc3c40acb60 con 0x7fc3c4142690 2024-03-27T03:11:06.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.619+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3c4131c30 con 0x7fc3c4142690 2024-03-27T03:11:06.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.619+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc3bc001e60 con 0x7fc3c4142690 2024-03-27T03:11:06.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:06.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:11:06.625 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 10 monclient(hunting): dump: 2024-03-27T03:11:06.625 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2234675823 0 0) 0x7fc3b0004eb0 con 0x7fc3c41422b0 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3c40acb60 con 0x7fc3c41422b0 2024-03-27T03:11:06.626 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc3b80017d0 con 0x7fc3c41422b0 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 10 monclient(hunting): dump: 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 692501278 0 0) 0x7fc3c4131c30 con 0x7fc3c4142690 2024-03-27T03:11:06.627 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 >> v1:172.21.15.153:6790/0 conn(0x7fc3c41422b0 legacy=0x7fc3c412e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.620+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 >> v1:172.21.15.153:6789/0 conn(0x7fc3c40b17d0 legacy=0x7fc3c40ab820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.621+0000 7fc3c0ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.621+0000 7fc3c0ff9640 1 monclient: found mon.b 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.621+0000 7fc3c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.621+0000 7fc3c0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.621+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c40ace40 con 0x7fc3c4142690 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.621+0000 7fc3c0ff9640 10 monclient: _finish_auth 0 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.621+0000 7fc3c0ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.621+0000 7fc3c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.621+0000 7fc3ca684640 5 monclient: authenticate success, global_id 4438 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.621+0000 7fc3ca684640 10 monclient: _renew_subs 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.621+0000 7fc3ca684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.621+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c41318c0 con 0x7fc3c4142690 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.622+0000 7fc3ca684640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:11:06.628 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.622+0000 7fc3ca684640 10 monclient: _renew_subs 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.622+0000 7fc3ca684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.622+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c40b6ad0 con 0x7fc3c4142690 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.622+0000 7fc3ca684640 1 librados: init done 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.623+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3bc001fc0 con 0x7fc3c4142690 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.623+0000 7fc3c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.623+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc3bc003dc0 con 0x7fc3c4142690 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.623+0000 7fc3c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.623+0000 7fc3c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.623+0000 7fc3c0ff9640 10 monclient: dump: 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:06.629 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.623+0000 7fc3c27fc640 4 set_mon_vals no callback set 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.624+0000 7fc3ca684640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.624+0000 7fc3ca684640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.624+0000 7fc3ca684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.624+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a4005080 con 0x7fc3c4142690 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.625+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fc3bc019d90 con 0x7fc3c4142690 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.626+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fc3bc083bc0 con 0x7fc3c4142690 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.626+0000 7fc3c0ff9640 10 client.4438.objecter ms_dispatch 0x7fc3c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.626+0000 7fc3c0ff9640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.626+0000 7fc3c0ff9640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.626+0000 7fc3c0ff9640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-27T03:11:06.630 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.626+0000 7fc3c3fff640 10 monclient: get_auth_request con 0x7fc3b006dda0 auth_method 0 2024-03-27T03:11:06.631 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.627+0000 7fc3c0ff9640 10 client.4438.objecter ms_handle_connect 0x7fc3b006dda0 2024-03-27T03:11:06.631 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.636+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3bc04f4a0 con 0x7fc3c4142690 2024-03-27T03:11:06.631 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.636+0000 7fc3c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:06.631 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.636+0000 7fc3c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:11:06.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.956+0000 7fc3ca684640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:06.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.956+0000 7fc3ca684640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:06.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.956+0000 7fc3ca684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:06.949 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.956+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3a4005330 con 0x7fc3c4142690 2024-03-27T03:11:06.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.957+0000 7fc3c0ff9640 1 -- 172.21.15.153:0/2206030102 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fc3bc0513d0 con 0x7fc3c4142690 2024-03-27T03:11:06.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.957+0000 7fc3c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:06.950 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.957+0000 7fc3c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:11:06.951 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:11:06.951 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:11:06.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.961+0000 7fc3ca684640 10 librados: watch_flush enter 2024-03-27T03:11:06.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.961+0000 7fc3ca684640 10 librados: watch_flush exit 2024-03-27T03:11:06.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.961+0000 7fc3ca684640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-27T03:11:06.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.961+0000 7fc3ca684640 10 client.4438.objecter successfully canceled tick 2024-03-27T03:11:06.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.962+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 >> v1:172.21.15.153:6816/2637937781 conn(0x7fc3b006dda0 legacy=0x7fc3b0070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:06.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.962+0000 7fc3ca684640 10 monclient: shutdown 2024-03-27T03:11:06.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.962+0000 7fc3ca684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:06.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.962+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 >> v1:172.21.15.196:6789/0 conn(0x7fc3c4142690 legacy=0x7fc3c412d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:06.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.962+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 shutdown_connections 2024-03-27T03:11:06.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.962+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 >> 172.21.15.153:0/2206030102 conn(0x7fc3c4066a00 msgr2=0x7fc3c413c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:06.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.962+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 shutdown_connections 2024-03-27T03:11:06.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.962+0000 7fc3ca684640 1 -- 172.21.15.153:0/2206030102 wait complete. 2024-03-27T03:11:06.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:06.962+0000 7fc3ca684640 1 librados: shutdown 2024-03-27T03:11:06.969 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:11:13.776 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:11:13.915 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.922+0000 7fc1257e6640 10 monclient: get_monmap_and_config 2024-03-27T03:11:13.915 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.922+0000 7fc1257e6640 10 monclient: build_initial_monmap 2024-03-27T03:11:13.915 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.922+0000 7fc1257e6640 10 monclient: monmap: 2024-03-27T03:11:13.915 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:13.915 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:13.915 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:11:13.923049+0000 2024-03-27T03:11:13.915 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:11:13.923049+0000 2024-03-27T03:11:13.915 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:13.915 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:13.916 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:13.916 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:11:13.916 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:11:13.916 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:13.917 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.922+0000 7fc1257e6640 1 Processor -- start 2024-03-27T03:11:13.917 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.923+0000 7fc1257e6640 1 -- start start 2024-03-27T03:11:13.917 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.923+0000 7fc1257e6640 10 monclient: init 2024-03-27T03:11:13.917 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.923+0000 7fc1257e6640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:13.917 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.923+0000 7fc1257e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:11:13.917 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.923+0000 7fc1257e6640 10 monclient(hunting): picked mon.noname-b con 0x7fc1200b0770 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.923+0000 7fc1257e6640 10 monclient(hunting): picked mon.noname-c con 0x7fc12014a700 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.923+0000 7fc1257e6640 10 monclient(hunting): picked mon.noname-a con 0x7fc1200ac5a0 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.923+0000 7fc1257e6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1200aba70 con 0x7fc1200ac5a0 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.923+0000 7fc1257e6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1200aecb0 con 0x7fc12014a700 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.923+0000 7fc1257e6640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1200af150 con 0x7fc1200b0770 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.923+0000 7fc1257e6640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.923+0000 7fc1257e6640 10 monclient(hunting): authenticate will time out at 1634.229496s 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.924+0000 7fc11effd640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fc1200b0770 0x7fc120149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:41328/0 (socket says 172.21.15.153:41328) 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.924+0000 7fc11effd640 1 -- 172.21.15.153:0/1351037805 learned_addr learned my addr 172.21.15.153:0/1351037805 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.924+0000 7fc11dffb640 1 -- 172.21.15.153:0/1351037805 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327132992 0 0) 0x7fc1200af150 con 0x7fc1200b0770 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.924+0000 7fc11dffb640 10 monclient(hunting): _negotiate 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.924+0000 7fc11dffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.924+0000 7fc11dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:13.918 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.924+0000 7fc11dffb640 10 monclient(hunting): my global_id is 4463 2024-03-27T03:11:13.919 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.924+0000 7fc11dffb640 1 -- 172.21.15.153:0/1351037805 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc108003520 con 0x7fc1200b0770 2024-03-27T03:11:13.919 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.924+0000 7fc11dffb640 1 -- 172.21.15.153:0/1351037805 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2927845620 0 0) 0x7fc108003520 con 0x7fc1200b0770 2024-03-27T03:11:13.919 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.924+0000 7fc11dffb640 1 -- 172.21.15.153:0/1351037805 >> v1:172.21.15.196:6789/0 conn(0x7fc12014a700 legacy=0x7fc1200abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:11:13.919 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.924+0000 7fc11dffb640 1 -- 172.21.15.153:0/1351037805 >> v1:172.21.15.153:6789/0 conn(0x7fc1200ac5a0 legacy=0x7fc120151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:11:13.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc11dffb640 10 monclient: _finish_hunting 0 2024-03-27T03:11:13.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc11dffb640 1 monclient: found mon.noname-b 2024-03-27T03:11:13.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc11dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:13.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc11dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:11:13.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc11dffb640 1 -- 172.21.15.153:0/1351037805 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1200ac9d0 con 0x7fc1200b0770 2024-03-27T03:11:13.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc11dffb640 10 monclient: _finish_auth 0 2024-03-27T03:11:13.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc11dffb640 10 monclient: _check_auth_tickets 2024-03-27T03:11:13.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc11dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:13.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc11dffb640 1 -- 172.21.15.153:0/1351037805 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc110001690 con 0x7fc1200b0770 2024-03-27T03:11:13.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc11dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:13.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc11dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:11:13.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc11dffb640 10 monclient: dump: 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc1257e6640 5 monclient: authenticate success, global_id 4463 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.926+0000 7fc1257e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.927+0000 7fc11dffb640 1 -- 172.21.15.153:0/1351037805 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1100018f0 con 0x7fc1200b0770 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.927+0000 7fc11dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:13.921 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.927+0000 7fc11dffb640 1 -- 172.21.15.153:0/1351037805 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc110001d40 con 0x7fc1200b0770 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.927+0000 7fc11dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.927+0000 7fc11dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.927+0000 7fc11dffb640 10 monclient: dump: 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.927+0000 7fc1257e6640 10 monclient: get_monmap_and_config success 2024-03-27T03:11:13.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.927+0000 7fc1257e6640 4 set_mon_vals no callback set 2024-03-27T03:11:13.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.928+0000 7fc1257e6640 10 monclient: shutdown 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.928+0000 7fc1257e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.928+0000 7fc1257e6640 1 -- 172.21.15.153:0/1351037805 >> v1:172.21.15.153:6790/0 conn(0x7fc1200b0770 legacy=0x7fc120149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.928+0000 7fc1257e6640 1 -- 172.21.15.153:0/1351037805 shutdown_connections 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.928+0000 7fc1257e6640 1 -- 172.21.15.153:0/1351037805 >> 172.21.15.153:0/1351037805 conn(0x7fc12006b660 msgr2=0x7fc12006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.928+0000 7fc1257e6640 1 -- 172.21.15.153:0/1351037805 shutdown_connections 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.928+0000 7fc1257e6640 1 -- 172.21.15.153:0/1351037805 wait complete. 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.929+0000 7fc1257e6640 10 monclient: build_initial_monmap 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.929+0000 7fc1257e6640 10 monclient: monmap: 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:13.924 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:13.925 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:11:13.925 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:11:13.925 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:13.932 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.929+0000 7fc1257e6640 1 librados: starting msgr at 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.929+0000 7fc1257e6640 1 librados: starting objecter 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.930+0000 7fc1257e6640 1 Processor -- start 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.930+0000 7fc1257e6640 1 -- start start 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.930+0000 7fc1257e6640 1 librados: setting wanted keys 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.930+0000 7fc1257e6640 1 librados: calling monclient init 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.930+0000 7fc1257e6640 10 monclient: init 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.930+0000 7fc1257e6640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.930+0000 7fc1257e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1257e6640 10 monclient(hunting): picked mon.noname-c con 0x7fc1200ac5a0 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1257e6640 10 monclient(hunting): picked mon.noname-a con 0x7fc1200b0770 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc11effd640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fc1200ac5a0 0x7fc1201ded40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:42136/0 (socket says 172.21.15.153:42136) 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1257e6640 10 monclient(hunting): picked mon.noname-b con 0x7fc12014a700 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc11effd640 1 -- 172.21.15.153:0/1455735170 learned_addr learned my addr 172.21.15.153:0/1455735170 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:13.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1257e6640 1 -- 172.21.15.153:0/1455735170 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1201df6a0 con 0x7fc1200b0770 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1257e6640 1 -- 172.21.15.153:0/1455735170 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc120145830 con 0x7fc12014a700 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1257e6640 1 -- 172.21.15.153:0/1455735170 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1201459c0 con 0x7fc1200ac5a0 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1257e6640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1257e6640 10 monclient(hunting): authenticate will time out at 1634.236915s 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1077fe640 10 client.?.objecter ms_handle_connect 0x7fc1200ac5a0 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1077fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1077fe640 10 client.?.objecter ms_handle_connect 0x7fc12014a700 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1077fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354308361 0 0) 0x7fc1201459c0 con 0x7fc1200ac5a0 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1077fe640 10 monclient(hunting): _negotiate 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1077fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1077fe640 10 monclient(hunting): my global_id is 4466 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.931+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc10c004eb0 con 0x7fc1200ac5a0 2024-03-27T03:11:13.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4043382992 0 0) 0x7fc10c004eb0 con 0x7fc1200ac5a0 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1201459c0 con 0x7fc1200ac5a0 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc110001650 con 0x7fc1200ac5a0 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 10 monclient(hunting): dump: 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:13.935 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966853135 0 0) 0x7fc120145830 con 0x7fc12014a700 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 10 monclient(hunting): _negotiate 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 10 monclient(hunting): my global_id is 4441 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc10c004eb0 con 0x7fc12014a700 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3803698985 0 0) 0x7fc1201459c0 con 0x7fc1200ac5a0 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 >> v1:172.21.15.196:6789/0 conn(0x7fc12014a700 legacy=0x7fc1201db460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.932+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 >> v1:172.21.15.153:6789/0 conn(0x7fc1200b0770 legacy=0x7fc120144ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1077fe640 10 monclient: _finish_hunting 0 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1077fe640 1 monclient: found mon.c 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1200703b0 con 0x7fc1200ac5a0 2024-03-27T03:11:13.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1077fe640 10 monclient: _finish_auth 0 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1077fe640 10 monclient: _check_auth_tickets 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1257e6640 5 monclient: authenticate success, global_id 4466 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1257e6640 10 monclient: _renew_subs 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1257e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1257e6640 1 -- 172.21.15.153:0/1455735170 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1200705a0 con 0x7fc1200ac5a0 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1257e6640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1257e6640 10 monclient: _renew_subs 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1257e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1257e6640 1 -- 172.21.15.153:0/1455735170 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1201453c0 con 0x7fc1200ac5a0 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.934+0000 7fc1257e6640 1 librados: init done 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.936+0000 7fc1057fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.936+0000 7fc1057fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.936+0000 7fc1057fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.936+0000 7fc1057fa640 1 -- 172.21.15.153:0/1455735170 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1200ab560 con 0x7fc1200ac5a0 2024-03-27T03:11:13.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.936+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc110004c70 con 0x7fc1200ac5a0 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.936+0000 7fc1077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.936+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fc110005000 con 0x7fc1200ac5a0 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.936+0000 7fc1077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.936+0000 7fc1077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.936+0000 7fc1077fe640 10 monclient: dump: 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:13.938 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:13.939 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:13.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.936+0000 7fc11d7fa640 4 set_mon_vals no callback set 2024-03-27T03:11:13.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.937+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fc110005220 con 0x7fc1200ac5a0 2024-03-27T03:11:13.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.938+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fc110084d50 con 0x7fc1200ac5a0 2024-03-27T03:11:13.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.938+0000 7fc1077fe640 10 client.4466.objecter ms_dispatch 0x7fc120000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:11:13.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.938+0000 7fc1077fe640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:11:13.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.938+0000 7fc1077fe640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:11:13.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.938+0000 7fc1077fe640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-27T03:11:13.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.938+0000 7fc11e7fc640 10 monclient: get_auth_request con 0x7fc10c06dc60 auth_method 0 2024-03-27T03:11:13.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.938+0000 7fc1077fe640 10 client.4466.objecter ms_handle_connect 0x7fc10c06dc60 2024-03-27T03:11:13.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.945+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc110050630 con 0x7fc1200ac5a0 2024-03-27T03:11:13.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.945+0000 7fc1077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:13.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:13.945+0000 7fc1077fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:11:14.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.250+0000 7fc1057fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:14.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.250+0000 7fc1057fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:14.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.250+0000 7fc1057fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:11:14.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.250+0000 7fc1057fa640 1 -- 172.21.15.153:0/1455735170 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1201df450 con 0x7fc1200ac5a0 2024-03-27T03:11:14.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.251+0000 7fc1077fe640 1 -- 172.21.15.153:0/1455735170 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fc110052560 con 0x7fc1200ac5a0 2024-03-27T03:11:14.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.251+0000 7fc1077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:14.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.251+0000 7fc1077fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:11:14.244 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:11:14.244 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:11:14.247 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.254+0000 7fc1057fa640 10 librados: watch_flush enter 2024-03-27T03:11:14.247 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.254+0000 7fc1057fa640 10 librados: watch_flush exit 2024-03-27T03:11:14.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.254+0000 7fc1057fa640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-27T03:11:14.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.254+0000 7fc1057fa640 10 client.4466.objecter successfully canceled tick 2024-03-27T03:11:14.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.254+0000 7fc1057fa640 1 -- 172.21.15.153:0/1455735170 >> v1:172.21.15.153:6816/2637937781 conn(0x7fc10c06dc60 legacy=0x7fc10c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:14.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.254+0000 7fc1057fa640 10 monclient: shutdown 2024-03-27T03:11:14.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.254+0000 7fc1057fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:14.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.254+0000 7fc1057fa640 1 -- 172.21.15.153:0/1455735170 >> v1:172.21.15.153:6790/0 conn(0x7fc1200ac5a0 legacy=0x7fc1201ded40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:14.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.254+0000 7fc1057fa640 1 -- 172.21.15.153:0/1455735170 shutdown_connections 2024-03-27T03:11:14.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.254+0000 7fc1057fa640 1 -- 172.21.15.153:0/1455735170 >> 172.21.15.153:0/1455735170 conn(0x7fc12006b660 msgr2=0x7fc1200b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:14.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.255+0000 7fc1057fa640 1 -- 172.21.15.153:0/1455735170 shutdown_connections 2024-03-27T03:11:14.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.255+0000 7fc1057fa640 1 -- 172.21.15.153:0/1455735170 wait complete. 2024-03-27T03:11:14.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:14.255+0000 7fc1057fa640 1 librados: shutdown 2024-03-27T03:11:14.263 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:11:21.070 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:11:21.216 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.222+0000 7fe4e7fff640 10 monclient: get_monmap_and_config 2024-03-27T03:11:21.216 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.222+0000 7fe4e7fff640 10 monclient: build_initial_monmap 2024-03-27T03:11:21.216 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.222+0000 7fe4e7fff640 10 monclient: monmap: 2024-03-27T03:11:21.217 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:21.217 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:21.217 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:11:21.223540+0000 2024-03-27T03:11:21.217 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:11:21.223540+0000 2024-03-27T03:11:21.217 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:21.217 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:21.217 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:21.218 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:11:21.218 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:11:21.218 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:21.218 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.223+0000 7fe4e7fff640 1 Processor -- start 2024-03-27T03:11:21.219 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.223+0000 7fe4e7fff640 1 -- start start 2024-03-27T03:11:21.219 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.223+0000 7fe4e7fff640 10 monclient: init 2024-03-27T03:11:21.219 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.224+0000 7fe4e7fff640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:21.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.224+0000 7fe4e7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:11:21.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.224+0000 7fe4e7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fe4e0036a90 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:21.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.224+0000 7fe4e7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fe4e00352e0 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:21.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.224+0000 7fe4e7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fe4e002ccf0 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:21.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.224+0000 7fe4e7fff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e002d3d0 con 0x7fe4e002ccf0 2024-03-27T03:11:21.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.224+0000 7fe4e7fff640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e002e370 con 0x7fe4e0036a90 2024-03-27T03:11:21.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.224+0000 7fe4e7fff640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e00c30c0 con 0x7fe4e00352e0 2024-03-27T03:11:21.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.224+0000 7fe4e7fff640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:21.221 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.224+0000 7fe4e7fff640 10 monclient(hunting): authenticate will time out at 1641.530177s 2024-03-27T03:11:21.221 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.225+0000 7fe4e6ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe4e0036a90 0x7fe4e00c29b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:54164/0 (socket says 172.21.15.153:54164) 2024-03-27T03:11:21.221 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.225+0000 7fe4e6ffd640 1 -- 172.21.15.153:0/2982334540 learned_addr learned my addr 172.21.15.153:0/2982334540 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:21.224 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.227+0000 7fe4e5ffb640 1 -- 172.21.15.153:0/2982334540 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104060171 0 0) 0x7fe4e002e370 con 0x7fe4e0036a90 2024-03-27T03:11:21.224 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:11:21.224 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:21.225 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:21.225 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient(hunting): my global_id is 4444 2024-03-27T03:11:21.225 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 1 -- 172.21.15.153:0/2982334540 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d0003520 con 0x7fe4e0036a90 2024-03-27T03:11:21.225 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 1 -- 172.21.15.153:0/2982334540 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918308338 0 0) 0x7fe4e002d3d0 con 0x7fe4e002ccf0 2024-03-27T03:11:21.225 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:11:21.225 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:21.225 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:21.226 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient(hunting): my global_id is 4413 2024-03-27T03:11:21.226 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 1 -- 172.21.15.153:0/2982334540 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e002e370 con 0x7fe4e002ccf0 2024-03-27T03:11:21.226 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 1 -- 172.21.15.153:0/2982334540 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3595135523 0 0) 0x7fe4e002e370 con 0x7fe4e002ccf0 2024-03-27T03:11:21.226 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 1 -- 172.21.15.153:0/2982334540 >> v1:172.21.15.153:6790/0 conn(0x7fe4e00352e0 legacy=0x7fe4e00c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:21.226 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 1 -- 172.21.15.153:0/2982334540 >> v1:172.21.15.196:6789/0 conn(0x7fe4e0036a90 legacy=0x7fe4e00c29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:21.226 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:11:21.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 1 monclient: found mon.noname-a 2024-03-27T03:11:21.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:21.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:11:21.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 1 -- 172.21.15.153:0/2982334540 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e00c3250 con 0x7fe4e002ccf0 2024-03-27T03:11:21.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient: _finish_auth 0 2024-03-27T03:11:21.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:11:21.227 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:21.228 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e7fff640 5 monclient: authenticate success, global_id 4413 2024-03-27T03:11:21.228 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:11:21.228 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 1 -- 172.21.15.153:0/2982334540 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe4e804f730 con 0x7fe4e002ccf0 2024-03-27T03:11:21.228 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.228+0000 7fe4e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:21.228 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.229+0000 7fe4e5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:11:21.228 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.229+0000 7fe4e5ffb640 10 monclient: dump: 2024-03-27T03:11:21.228 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:21.228 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:21.229 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:21.229 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:21.229 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:21.229 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:21.229 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:21.229 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:21.229 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:21.230 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:21.230 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.229+0000 7fe4e7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:11:21.230 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.229+0000 7fe4e5ffb640 1 -- 172.21.15.153:0/2982334540 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4e8075f10 con 0x7fe4e002ccf0 2024-03-27T03:11:21.230 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.229+0000 7fe4e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:21.230 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.229+0000 7fe4e7fff640 10 monclient: get_monmap_and_config success 2024-03-27T03:11:21.230 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.229+0000 7fe4e7fff640 4 set_mon_vals no callback set 2024-03-27T03:11:21.230 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.230+0000 7fe4e7fff640 10 monclient: shutdown 2024-03-27T03:11:21.230 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.230+0000 7fe4e7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:21.231 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.230+0000 7fe4e7fff640 1 -- 172.21.15.153:0/2982334540 >> v1:172.21.15.153:6789/0 conn(0x7fe4e002ccf0 legacy=0x7fe4e00c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:21.231 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.231+0000 7fe4e7fff640 1 -- 172.21.15.153:0/2982334540 shutdown_connections 2024-03-27T03:11:21.231 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.231+0000 7fe4e7fff640 1 -- 172.21.15.153:0/2982334540 >> 172.21.15.153:0/2982334540 conn(0x7fe4e00058f0 msgr2=0x7fe4e0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:21.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.233+0000 7fe4e7fff640 1 -- 172.21.15.153:0/2982334540 shutdown_connections 2024-03-27T03:11:21.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.234+0000 7fe4e7fff640 1 -- 172.21.15.153:0/2982334540 wait complete. 2024-03-27T03:11:21.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.235+0000 7fe4e7fff640 10 monclient: build_initial_monmap 2024-03-27T03:11:21.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.235+0000 7fe4e7fff640 10 monclient: monmap: 2024-03-27T03:11:21.242 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:21.243 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:11:21.243 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:11:21.243 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:11:21.243 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:21.243 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:21.243 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:21.243 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:11:21.243 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:11:21.244 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:21.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.236+0000 7fe4e7fff640 1 librados: starting msgr at 2024-03-27T03:11:21.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.236+0000 7fe4e7fff640 1 librados: starting objecter 2024-03-27T03:11:21.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.236+0000 7fe4e7fff640 1 Processor -- start 2024-03-27T03:11:21.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.236+0000 7fe4e7fff640 1 -- start start 2024-03-27T03:11:21.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.236+0000 7fe4e7fff640 1 librados: setting wanted keys 2024-03-27T03:11:21.244 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.236+0000 7fe4e7fff640 1 librados: calling monclient init 2024-03-27T03:11:21.245 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.236+0000 7fe4e7fff640 10 monclient: init 2024-03-27T03:11:21.245 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.237+0000 7fe4e7fff640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:21.245 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.237+0000 7fe4e7fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:11:21.245 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.238+0000 7fe4e7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fe4e002ccf0 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:21.245 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.238+0000 7fe4e7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fe4e00352e0 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:21.245 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.240+0000 7fe4e7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fe4e0036a90 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:21.245 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.240+0000 7fe4e7fff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e015b670 con 0x7fe4e002ccf0 2024-03-27T03:11:21.245 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.240+0000 7fe4e7fff640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e015c4f0 con 0x7fe4e00352e0 2024-03-27T03:11:21.246 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.240+0000 7fe4e7fff640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e015b8c0 con 0x7fe4e0036a90 2024-03-27T03:11:21.246 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.240+0000 7fe4e7fff640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:21.246 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.240+0000 7fe4e7fff640 10 monclient(hunting): authenticate will time out at 1641.546046s 2024-03-27T03:11:21.246 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.241+0000 7fe4e6ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe4e002ccf0 0x7fe4e015bde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:46499/0 (socket says 172.21.15.153:46499) 2024-03-27T03:11:21.246 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.241+0000 7fe4e6ffd640 1 -- 172.21.15.153:0/3700673198 learned_addr learned my addr 172.21.15.153:0/3700673198 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:21.246 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.241+0000 7fe4cf7fe640 10 client.?.objecter ms_handle_connect 0x7fe4e002ccf0 2024-03-27T03:11:21.247 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.241+0000 7fe4cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:21.247 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.241+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634812140 0 0) 0x7fe4e015b670 con 0x7fe4e002ccf0 2024-03-27T03:11:21.247 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.241+0000 7fe4cf7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:11:21.247 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.241+0000 7fe4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:21.247 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.241+0000 7fe4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:21.247 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.241+0000 7fe4cf7fe640 10 monclient(hunting): my global_id is 4416 2024-03-27T03:11:21.247 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.241+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d4004eb0 con 0x7fe4e002ccf0 2024-03-27T03:11:21.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.242+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1765534913 0 0) 0x7fe4d4004eb0 con 0x7fe4e002ccf0 2024-03-27T03:11:21.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.242+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4d4005300 con 0x7fe4e002ccf0 2024-03-27T03:11:21.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.242+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe4dc001830 con 0x7fe4e002ccf0 2024-03-27T03:11:21.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.242+0000 7fe4cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:21.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.242+0000 7fe4cf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:11:21.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.242+0000 7fe4cf7fe640 10 monclient(hunting): dump: 2024-03-27T03:11:21.248 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:21.248 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:21.249 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:21.249 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:21.249 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:21.249 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:21.249 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:21.249 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:21.249 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:21.250 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:21.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.242+0000 7fe4cf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:11:21.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.242+0000 7fe4cf7fe640 10 client.?.objecter ms_handle_connect 0x7fe4e00352e0 2024-03-27T03:11:21.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.242+0000 7fe4cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:21.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.242+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765383731 0 0) 0x7fe4e015c4f0 con 0x7fe4e00352e0 2024-03-27T03:11:21.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:11:21.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:21.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:21.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 10 monclient(hunting): my global_id is 4447 2024-03-27T03:11:21.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d40058a0 con 0x7fe4e00352e0 2024-03-27T03:11:21.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2357282599 0 0) 0x7fe4d4005300 con 0x7fe4e002ccf0 2024-03-27T03:11:21.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 >> v1:172.21.15.153:6790/0 conn(0x7fe4e0036a90 legacy=0x7fe4e00c1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:21.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 >> v1:172.21.15.196:6789/0 conn(0x7fe4e00352e0 legacy=0x7fe4e00b9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:21.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 10 monclient: _finish_hunting 0 2024-03-27T03:11:21.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 1 monclient: found mon.a 2024-03-27T03:11:21.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:21.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:21.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e00c0bc0 con 0x7fe4e002ccf0 2024-03-27T03:11:21.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 10 monclient: _finish_auth 0 2024-03-27T03:11:21.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 10 monclient: _check_auth_tickets 2024-03-27T03:11:21.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:21.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4e7fff640 5 monclient: authenticate success, global_id 4416 2024-03-27T03:11:21.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4e7fff640 10 monclient: _renew_subs 2024-03-27T03:11:21.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4e7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:21.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4e7fff640 1 -- 172.21.15.153:0/3700673198 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e00b9790 con 0x7fe4e002ccf0 2024-03-27T03:11:21.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4e7fff640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:11:21.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4e7fff640 10 monclient: _renew_subs 2024-03-27T03:11:21.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4e7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:21.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4e7fff640 1 -- 172.21.15.153:0/3700673198 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e00c2030 con 0x7fe4e002ccf0 2024-03-27T03:11:21.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4e7fff640 1 librados: init done 2024-03-27T03:11:21.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4dc001d60 con 0x7fe4e002ccf0 2024-03-27T03:11:21.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:21.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe4dc001830 con 0x7fe4e002ccf0 2024-03-27T03:11:21.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:21.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:11:21.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4e4ff9640 4 set_mon_vals no callback set 2024-03-27T03:11:21.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.243+0000 7fe4cf7fe640 10 monclient: dump: 2024-03-27T03:11:21.255 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:21.255 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:21.255 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:21.255 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:21.255 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:21.255 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:21.255 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:21.255 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:21.256 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:21.256 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:21.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.245+0000 7fe4e7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:21.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.245+0000 7fe4e7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:21.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.245+0000 7fe4e7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:21.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.245+0000 7fe4e7fff640 1 -- 172.21.15.153:0/3700673198 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c8005080 con 0x7fe4e002ccf0 2024-03-27T03:11:21.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.245+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fe4dc01a6d0 con 0x7fe4e002ccf0 2024-03-27T03:11:21.256 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.246+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fe4dc0544e0 con 0x7fe4e002ccf0 2024-03-27T03:11:21.257 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.246+0000 7fe4cf7fe640 10 client.4416.objecter ms_dispatch 0x7fe4e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:11:21.257 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.246+0000 7fe4cf7fe640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:11:21.257 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.246+0000 7fe4cf7fe640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:11:21.257 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.246+0000 7fe4cf7fe640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-27T03:11:21.257 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.246+0000 7fe4e67fc640 10 monclient: get_auth_request con 0x7fe4d406e300 auth_method 0 2024-03-27T03:11:21.257 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.247+0000 7fe4cf7fe640 10 client.4416.objecter ms_handle_connect 0x7fe4d406e300 2024-03-27T03:11:21.258 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.255+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4dc050780 con 0x7fe4e002ccf0 2024-03-27T03:11:21.258 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.255+0000 7fe4cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:21.258 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.255+0000 7fe4cf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:11:21.566 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.573+0000 7fe4e7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:21.566 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.573+0000 7fe4e7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:21.566 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.573+0000 7fe4e7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:21.567 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.573+0000 7fe4e7fff640 1 -- 172.21.15.153:0/3700673198 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4c8005330 con 0x7fe4e002ccf0 2024-03-27T03:11:21.568 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.576+0000 7fe4cf7fe640 1 -- 172.21.15.153:0/3700673198 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fe4dc050920 con 0x7fe4e002ccf0 2024-03-27T03:11:21.569 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.576+0000 7fe4cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:21.569 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.576+0000 7fe4cf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:11:21.570 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:11:21.570 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:11:21.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.579+0000 7fe4e7fff640 10 librados: watch_flush enter 2024-03-27T03:11:21.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.579+0000 7fe4e7fff640 10 librados: watch_flush exit 2024-03-27T03:11:21.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.579+0000 7fe4e7fff640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-27T03:11:21.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.579+0000 7fe4e7fff640 10 client.4416.objecter successfully canceled tick 2024-03-27T03:11:21.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.579+0000 7fe4e7fff640 1 -- 172.21.15.153:0/3700673198 >> v1:172.21.15.153:6816/2637937781 conn(0x7fe4d406e300 legacy=0x7fe4d40707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:21.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.580+0000 7fe4e7fff640 10 monclient: shutdown 2024-03-27T03:11:21.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.580+0000 7fe4e7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:21.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.580+0000 7fe4e7fff640 1 -- 172.21.15.153:0/3700673198 >> v1:172.21.15.153:6789/0 conn(0x7fe4e002ccf0 legacy=0x7fe4e015bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:21.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.580+0000 7fe4e7fff640 1 -- 172.21.15.153:0/3700673198 shutdown_connections 2024-03-27T03:11:21.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.580+0000 7fe4e7fff640 1 -- 172.21.15.153:0/3700673198 >> 172.21.15.153:0/3700673198 conn(0x7fe4e00058f0 msgr2=0x7fe4e00bf1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:21.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.580+0000 7fe4e7fff640 1 -- 172.21.15.153:0/3700673198 shutdown_connections 2024-03-27T03:11:21.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.580+0000 7fe4e7fff640 1 -- 172.21.15.153:0/3700673198 wait complete. 2024-03-27T03:11:21.574 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:21.580+0000 7fe4e7fff640 1 librados: shutdown 2024-03-27T03:11:21.592 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:11:28.399 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:11:28.558 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.565+0000 7f7d8300d640 10 monclient: get_monmap_and_config 2024-03-27T03:11:28.559 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.565+0000 7f7d8300d640 10 monclient: build_initial_monmap 2024-03-27T03:11:28.559 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.565+0000 7f7d8300d640 10 monclient: monmap: 2024-03-27T03:11:28.559 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:28.559 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:28.559 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:11:28.566733+0000 2024-03-27T03:11:28.559 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:11:28.566733+0000 2024-03-27T03:11:28.559 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:28.559 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:28.559 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:28.559 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:11:28.559 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:11:28.559 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:28.562 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.569+0000 7f7d8300d640 1 Processor -- start 2024-03-27T03:11:28.563 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.570+0000 7f7d8300d640 1 -- start start 2024-03-27T03:11:28.563 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.570+0000 7f7d8300d640 10 monclient: init 2024-03-27T03:11:28.565 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.571+0000 7f7d8300d640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:28.565 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.571+0000 7f7d8300d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:11:28.565 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.571+0000 7f7d8300d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d7c0b4790 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:28.565 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.571+0000 7f7d8300d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d7c0aea90 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:28.565 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.571+0000 7f7d8300d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d7c0b39f0 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:28.565 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.571+0000 7f7d8300d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7c0b0c60 con 0x7f7d7c0aea90 2024-03-27T03:11:28.565 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.572+0000 7f7d8300d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7c0af020 con 0x7f7d7c0b4790 2024-03-27T03:11:28.565 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.572+0000 7f7d8300d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7c13fbd0 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.565 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.572+0000 7f7d8300d640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:28.565 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.572+0000 7f7d8300d640 10 monclient(hunting): authenticate will time out at 1648.877774s 2024-03-27T03:11:28.568 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.575+0000 7f7d7bfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f7d7c0aea90 0x7f7d7c0af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39017/0 (socket says 172.21.15.153:39017) 2024-03-27T03:11:28.568 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.575+0000 7f7d7bfff640 1 -- 172.21.15.153:0/1814086789 learned_addr learned my addr 172.21.15.153:0/1814086789 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:28.571 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.578+0000 7f7d7b7fe640 1 -- 172.21.15.153:0/1814086789 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117922468 0 0) 0x7f7d7c0b0c60 con 0x7f7d7c0aea90 2024-03-27T03:11:28.571 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.578+0000 7f7d7b7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:11:28.571 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.578+0000 7f7d7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:28.571 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.578+0000 7f7d7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:28.571 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.578+0000 7f7d7b7fe640 10 monclient(hunting): my global_id is 4419 2024-03-27T03:11:28.571 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.578+0000 7f7d7b7fe640 1 -- 172.21.15.153:0/1814086789 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d68003520 con 0x7f7d7c0aea90 2024-03-27T03:11:28.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.579+0000 7f7d7b7fe640 1 -- 172.21.15.153:0/1814086789 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844506510 0 0) 0x7f7d7c13fbd0 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.579+0000 7f7d7b7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:11:28.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.579+0000 7f7d7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:28.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.579+0000 7f7d7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:28.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.579+0000 7f7d7b7fe640 10 monclient(hunting): my global_id is 4472 2024-03-27T03:11:28.573 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.579+0000 7f7d7b7fe640 1 -- 172.21.15.153:0/1814086789 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d7c0b0c60 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.574 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 1 -- 172.21.15.153:0/1814086789 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2007414673 0 0) 0x7f7d68003520 con 0x7f7d7c0aea90 2024-03-27T03:11:28.574 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 1 -- 172.21.15.153:0/1814086789 >> v1:172.21.15.153:6790/0 conn(0x7f7d7c0b39f0 legacy=0x7f7d7c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:28.574 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 1 -- 172.21.15.153:0/1814086789 >> v1:172.21.15.196:6789/0 conn(0x7f7d7c0b4790 legacy=0x7f7d7c0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:28.575 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 10 monclient: _finish_hunting 0 2024-03-27T03:11:28.575 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 1 monclient: found mon.noname-a 2024-03-27T03:11:28.575 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:28.575 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:11:28.575 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 1 -- 172.21.15.153:0/1814086789 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c13fd60 con 0x7f7d7c0aea90 2024-03-27T03:11:28.575 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 10 monclient: _finish_auth 0 2024-03-27T03:11:28.575 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 10 monclient: _check_auth_tickets 2024-03-27T03:11:28.575 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:28.575 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 1 -- 172.21.15.153:0/1814086789 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f7d64001690 con 0x7f7d7c0aea90 2024-03-27T03:11:28.575 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:28.575 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:11:28.576 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.580+0000 7f7d7b7fe640 10 monclient: dump: 2024-03-27T03:11:28.576 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:28.576 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:28.576 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:28.576 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:28.576 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:28.576 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:28.576 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:28.576 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:28.576 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:28.577 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:28.577 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.581+0000 7f7d7b7fe640 1 -- 172.21.15.153:0/1814086789 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d640017f0 con 0x7f7d7c0aea90 2024-03-27T03:11:28.577 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.581+0000 7f7d7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:28.577 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.581+0000 7f7d7b7fe640 1 -- 172.21.15.153:0/1814086789 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f7d64001c00 con 0x7f7d7c0aea90 2024-03-27T03:11:28.577 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.581+0000 7f7d7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:28.577 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.581+0000 7f7d7b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:11:28.577 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.581+0000 7f7d7b7fe640 10 monclient: dump: 2024-03-27T03:11:28.577 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:28.577 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:28.577 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:28.577 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:28.577 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:28.578 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:28.578 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:28.578 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:28.578 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:28.578 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:28.579 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.582+0000 7f7d8300d640 5 monclient: authenticate success, global_id 4419 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.582+0000 7f7d8300d640 10 monclient: get_monmap_and_config success 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.582+0000 7f7d8300d640 4 set_mon_vals no callback set 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.583+0000 7f7d8300d640 10 monclient: shutdown 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.583+0000 7f7d8300d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.583+0000 7f7d8300d640 1 -- 172.21.15.153:0/1814086789 >> v1:172.21.15.153:6789/0 conn(0x7f7d7c0aea90 legacy=0x7f7d7c0af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.583+0000 7f7d8300d640 1 -- 172.21.15.153:0/1814086789 shutdown_connections 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.583+0000 7f7d8300d640 1 -- 172.21.15.153:0/1814086789 >> 172.21.15.153:0/1814086789 conn(0x7f7d7c066a00 msgr2=0x7f7d7c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.583+0000 7f7d8300d640 1 -- 172.21.15.153:0/1814086789 shutdown_connections 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.583+0000 7f7d8300d640 1 -- 172.21.15.153:0/1814086789 wait complete. 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.584+0000 7f7d8300d640 10 monclient: build_initial_monmap 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.585+0000 7f7d8300d640 10 monclient: monmap: 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:28.580 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:11:28.581 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:11:28.581 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:11:28.581 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:28.581 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:28.581 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:28.581 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:11:28.581 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:11:28.581 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:28.581 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.585+0000 7f7d8300d640 1 librados: starting msgr at 2024-03-27T03:11:28.581 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.585+0000 7f7d8300d640 1 librados: starting objecter 2024-03-27T03:11:28.581 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.585+0000 7f7d8300d640 1 Processor -- start 2024-03-27T03:11:28.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.586+0000 7f7d8300d640 1 -- start start 2024-03-27T03:11:28.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.586+0000 7f7d8300d640 1 librados: setting wanted keys 2024-03-27T03:11:28.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.586+0000 7f7d8300d640 1 librados: calling monclient init 2024-03-27T03:11:28.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.586+0000 7f7d8300d640 10 monclient: init 2024-03-27T03:11:28.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.587+0000 7f7d8300d640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:28.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.587+0000 7f7d8300d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:11:28.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.587+0000 7f7d8300d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d7c0b39f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:28.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.587+0000 7f7d8300d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d7c0b4790 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:28.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.587+0000 7f7d8300d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d7c070820 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:28.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.587+0000 7f7d8300d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7c142470 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.587+0000 7f7d8300d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7c0aa450 con 0x7f7d7c0b4790 2024-03-27T03:11:28.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.587+0000 7f7d8300d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7c142050 con 0x7f7d7c070820 2024-03-27T03:11:28.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.587+0000 7f7d8300d640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:28.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.587+0000 7f7d8300d640 10 monclient(hunting): authenticate will time out at 1648.893158s 2024-03-27T03:11:28.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.591+0000 7f7d815aa640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f7d7c070820 0x7f7d7c1ddc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:50435/0 (socket says 172.21.15.153:50435) 2024-03-27T03:11:28.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.591+0000 7f7d815aa640 1 -- 172.21.15.153:0/388453199 learned_addr learned my addr 172.21.15.153:0/388453199 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:28.586 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.593+0000 7f7d797fa640 10 client.?.objecter ms_handle_connect 0x7f7d7c070820 2024-03-27T03:11:28.586 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.593+0000 7f7d797fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:28.586 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.593+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369172017 0 0) 0x7f7d7c142050 con 0x7f7d7c070820 2024-03-27T03:11:28.586 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.593+0000 7f7d797fa640 10 monclient(hunting): _negotiate 2024-03-27T03:11:28.586 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.593+0000 7f7d797fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:28.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.593+0000 7f7d797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:28.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.593+0000 7f7d797fa640 10 monclient(hunting): my global_id is 4475 2024-03-27T03:11:28.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.593+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d5c004eb0 con 0x7f7d7c070820 2024-03-27T03:11:28.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.594+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 537977480 0 0) 0x7f7d5c004eb0 con 0x7f7d7c070820 2024-03-27T03:11:28.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.594+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d7c142050 con 0x7f7d7c070820 2024-03-27T03:11:28.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.594+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f7d74001990 con 0x7f7d7c070820 2024-03-27T03:11:28.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.594+0000 7f7d797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:28.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.594+0000 7f7d797fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:11:28.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.594+0000 7f7d797fa640 10 monclient(hunting): dump: 2024-03-27T03:11:28.588 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:28.588 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:28.588 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:28.588 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:28.588 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:28.588 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:28.589 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:28.589 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:28.589 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:28.589 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:28.591 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.594+0000 7f7d797fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:11:28.591 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.595+0000 7f7d797fa640 10 client.?.objecter ms_handle_connect 0x7f7d7c0b4790 2024-03-27T03:11:28.591 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.595+0000 7f7d797fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:28.591 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.596+0000 7f7d797fa640 10 client.?.objecter ms_handle_connect 0x7f7d7c0b39f0 2024-03-27T03:11:28.591 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.596+0000 7f7d797fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:28.591 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.596+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205779125 0 0) 0x7f7d7c142470 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.591 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.596+0000 7f7d797fa640 10 monclient(hunting): _negotiate 2024-03-27T03:11:28.591 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.596+0000 7f7d797fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:28.591 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.596+0000 7f7d797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:28.591 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.596+0000 7f7d797fa640 10 monclient(hunting): my global_id is 4422 2024-03-27T03:11:28.591 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.596+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d5c004eb0 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1533817378 0 0) 0x7f7d5c004eb0 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d7c142470 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f7d6c001b10 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 10 monclient(hunting): dump: 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:28.592 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:28.593 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:28.593 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:28.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1827064647 0 0) 0x7f7d7c142470 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 >> v1:172.21.15.153:6790/0 conn(0x7f7d7c070820 legacy=0x7f7d7c1ddc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:28.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 >> v1:172.21.15.196:6789/0 conn(0x7f7d7c0b4790 legacy=0x7f7d7c141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:28.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 10 monclient: _finish_hunting 0 2024-03-27T03:11:28.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 1 monclient: found mon.a 2024-03-27T03:11:28.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:28.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:28.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c13ff70 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 10 monclient: _finish_auth 0 2024-03-27T03:11:28.593 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 10 monclient: _check_auth_tickets 2024-03-27T03:11:28.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:28.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d8300d640 5 monclient: authenticate success, global_id 4422 2024-03-27T03:11:28.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d8300d640 10 monclient: _renew_subs 2024-03-27T03:11:28.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d8300d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:28.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d8300d640 1 -- 172.21.15.153:0/388453199 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d7c1401c0 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d8300d640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:11:28.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d8300d640 10 monclient: _renew_subs 2024-03-27T03:11:28.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d8300d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:28.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d8300d640 1 -- 172.21.15.153:0/388453199 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d7c1df520 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d8300d640 1 librados: init done 2024-03-27T03:11:28.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d6c0020a0 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.595 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:28.595 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f7d6c003b10 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.595 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:28.595 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:11:28.595 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.597+0000 7f7d797fa640 10 monclient: dump: 2024-03-27T03:11:28.595 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:28.595 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:28.595 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:28.595 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:28.595 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:28.595 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:28.596 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:28.596 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:28.596 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:28.596 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:28.597 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.598+0000 7f7d7a7fc640 4 set_mon_vals no callback set 2024-03-27T03:11:28.597 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.600+0000 7f7d8300d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:28.597 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.600+0000 7f7d8300d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:28.597 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.600+0000 7f7d8300d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:28.597 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.600+0000 7f7d8300d640 1 -- 172.21.15.153:0/388453199 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d60005080 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.597 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.602+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f7d6c003c70 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.597 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.603+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f7d6c083a70 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.597 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.603+0000 7f7d7bfff640 10 monclient: get_auth_request con 0x7f7d5c06dfb0 auth_method 0 2024-03-27T03:11:28.597 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.603+0000 7f7d797fa640 10 client.4422.objecter ms_dispatch 0x7f7d7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:11:28.597 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.603+0000 7f7d797fa640 3 client.4422.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:11:28.597 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.603+0000 7f7d797fa640 3 client.4422.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:11:28.597 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.603+0000 7f7d797fa640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-27T03:11:28.598 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.603+0000 7f7d797fa640 10 client.4422.objecter ms_handle_connect 0x7f7d5c06dfb0 2024-03-27T03:11:28.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.611+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d6c04f350 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.611+0000 7f7d797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:28.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.611+0000 7f7d797fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:11:28.926 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.932+0000 7f7d8300d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:28.926 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.933+0000 7f7d8300d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:28.926 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.933+0000 7f7d8300d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:28.926 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.933+0000 7f7d8300d640 1 -- 172.21.15.153:0/388453199 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d60005330 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.926 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.933+0000 7f7d797fa640 1 -- 172.21.15.153:0/388453199 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f7d6c051280 con 0x7f7d7c0b39f0 2024-03-27T03:11:28.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.933+0000 7f7d797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:28.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.933+0000 7f7d797fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:11:28.929 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:11:28.930 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:11:28.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.942+0000 7f7d8300d640 10 librados: watch_flush enter 2024-03-27T03:11:28.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.942+0000 7f7d8300d640 10 librados: watch_flush exit 2024-03-27T03:11:28.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.942+0000 7f7d8300d640 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-27T03:11:28.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.942+0000 7f7d8300d640 10 client.4422.objecter successfully canceled tick 2024-03-27T03:11:28.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.943+0000 7f7d8300d640 1 -- 172.21.15.153:0/388453199 >> v1:172.21.15.153:6816/2637937781 conn(0x7f7d5c06dfb0 legacy=0x7f7d5c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:28.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.943+0000 7f7d8300d640 10 monclient: shutdown 2024-03-27T03:11:28.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.943+0000 7f7d8300d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:28.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.943+0000 7f7d8300d640 1 -- 172.21.15.153:0/388453199 >> v1:172.21.15.153:6789/0 conn(0x7f7d7c0b39f0 legacy=0x7f7d7c142900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:28.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.943+0000 7f7d8300d640 1 -- 172.21.15.153:0/388453199 shutdown_connections 2024-03-27T03:11:28.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.943+0000 7f7d8300d640 1 -- 172.21.15.153:0/388453199 >> 172.21.15.153:0/388453199 conn(0x7f7d7c066a00 msgr2=0x7f7d7c0b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:28.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.943+0000 7f7d8300d640 1 -- 172.21.15.153:0/388453199 shutdown_connections 2024-03-27T03:11:28.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.943+0000 7f7d8300d640 1 -- 172.21.15.153:0/388453199 wait complete. 2024-03-27T03:11:28.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:28.943+0000 7f7d8300d640 1 librados: shutdown 2024-03-27T03:11:28.954 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:11:35.761 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:11:35.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.927+0000 7f967911c640 10 monclient: get_monmap_and_config 2024-03-27T03:11:35.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.927+0000 7f967911c640 10 monclient: build_initial_monmap 2024-03-27T03:11:35.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.928+0000 7f967911c640 10 monclient: monmap: 2024-03-27T03:11:35.922 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:35.922 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:35.922 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:11:35.929952+0000 2024-03-27T03:11:35.922 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:11:35.929952+0000 2024-03-27T03:11:35.922 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:35.922 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:35.922 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:35.923 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:11:35.923 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:11:35.923 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:35.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.931+0000 7f967911c640 1 Processor -- start 2024-03-27T03:11:35.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.933+0000 7f967911c640 1 -- start start 2024-03-27T03:11:35.927 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.933+0000 7f967911c640 10 monclient: init 2024-03-27T03:11:35.928 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.934+0000 7f967911c640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:35.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.934+0000 7f967911c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:11:35.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.934+0000 7f967911c640 10 monclient(hunting): picked mon.noname-a con 0x7f96740b1f60 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:35.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.934+0000 7f967911c640 10 monclient(hunting): picked mon.noname-b con 0x7f9674071eb0 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:35.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.934+0000 7f967911c640 10 monclient(hunting): picked mon.noname-c con 0x7f9674143d10 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:35.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.934+0000 7f967911c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967406ef80 con 0x7f96740b1f60 2024-03-27T03:11:35.929 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.934+0000 7f967911c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967406f260 con 0x7f9674143d10 2024-03-27T03:11:35.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.934+0000 7f967911c640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96740b40c0 con 0x7f9674071eb0 2024-03-27T03:11:35.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.934+0000 7f967911c640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:35.930 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.934+0000 7f967911c640 10 monclient(hunting): authenticate will time out at 1656.241111s 2024-03-27T03:11:35.932 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.938+0000 7f967359e640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f9674143d10 0x7f9674146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:59975/0 (socket says 172.21.15.153:59975) 2024-03-27T03:11:35.932 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.938+0000 7f967359e640 1 -- 172.21.15.153:0/2513995526 learned_addr learned my addr 172.21.15.153:0/2513995526 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:35.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.939+0000 7f9671d9b640 1 -- 172.21.15.153:0/2513995526 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062148 0 0) 0x7f967406f260 con 0x7f9674143d10 2024-03-27T03:11:35.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.939+0000 7f9671d9b640 10 monclient(hunting): _negotiate 2024-03-27T03:11:35.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.939+0000 7f9671d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.939+0000 7f9671d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.939+0000 7f9671d9b640 10 monclient(hunting): my global_id is 4453 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.939+0000 7f9671d9b640 1 -- 172.21.15.153:0/2513995526 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f965c003520 con 0x7f9674143d10 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 1 -- 172.21.15.153:0/2513995526 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 23219781 0 0) 0x7f965c003520 con 0x7f9674143d10 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 1 -- 172.21.15.153:0/2513995526 >> v1:172.21.15.153:6790/0 conn(0x7f9674071eb0 legacy=0x7f9674146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 1 -- 172.21.15.153:0/2513995526 >> v1:172.21.15.153:6789/0 conn(0x7f96740b1f60 legacy=0x7f9674143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 10 monclient: _finish_hunting 0 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 1 monclient: found mon.noname-c 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 1 -- 172.21.15.153:0/2513995526 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9674155d70 con 0x7f9674143d10 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 10 monclient: _finish_auth 0 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 10 monclient: _check_auth_tickets 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f967911c640 5 monclient: authenticate success, global_id 4453 2024-03-27T03:11:35.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f967911c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 1 -- 172.21.15.153:0/2513995526 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f966c001a10 con 0x7f9674143d10 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 10 monclient: dump: 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:35.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f967911c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:11:35.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 1 -- 172.21.15.153:0/2513995526 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f966c001ca0 con 0x7f9674143d10 2024-03-27T03:11:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.940+0000 7f9671d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.941+0000 7f967911c640 10 monclient: get_monmap_and_config success 2024-03-27T03:11:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.941+0000 7f967911c640 4 set_mon_vals no callback set 2024-03-27T03:11:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.942+0000 7f967911c640 10 monclient: shutdown 2024-03-27T03:11:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.942+0000 7f967911c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.942+0000 7f967911c640 1 -- 172.21.15.153:0/2513995526 >> v1:172.21.15.196:6789/0 conn(0x7f9674143d10 legacy=0x7f9674146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.942+0000 7f967911c640 1 -- 172.21.15.153:0/2513995526 shutdown_connections 2024-03-27T03:11:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.942+0000 7f967911c640 1 -- 172.21.15.153:0/2513995526 >> 172.21.15.153:0/2513995526 conn(0x7f9674066a00 msgr2=0x7f9674066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.942+0000 7f967911c640 1 -- 172.21.15.153:0/2513995526 shutdown_connections 2024-03-27T03:11:35.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.942+0000 7f967911c640 1 -- 172.21.15.153:0/2513995526 wait complete. 2024-03-27T03:11:35.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.946+0000 7f967911c640 10 monclient: build_initial_monmap 2024-03-27T03:11:35.946 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.946+0000 7f967911c640 10 monclient: monmap: 2024-03-27T03:11:35.946 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:35.946 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:11:35.946 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:11:35.946 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:11:35.946 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:35.946 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:35.946 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:35.946 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.947+0000 7f967911c640 1 librados: starting msgr at 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.947+0000 7f967911c640 1 librados: starting objecter 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.947+0000 7f967911c640 1 Processor -- start 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.947+0000 7f967911c640 1 -- start start 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.947+0000 7f967911c640 1 librados: setting wanted keys 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.947+0000 7f967911c640 1 librados: calling monclient init 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.947+0000 7f967911c640 10 monclient: init 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.948+0000 7f967911c640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.948+0000 7f967911c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.948+0000 7f967911c640 10 monclient(hunting): picked mon.noname-a con 0x7f96740b1f60 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.948+0000 7f967911c640 10 monclient(hunting): picked mon.noname-c con 0x7f9674143d10 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.948+0000 7f967911c640 10 monclient(hunting): picked mon.noname-b con 0x7f9674154260 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.948+0000 7f967911c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96740b81a0 con 0x7f96740b1f60 2024-03-27T03:11:35.947 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.948+0000 7f967911c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96740b64f0 con 0x7f9674154260 2024-03-27T03:11:35.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.948+0000 7f967911c640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96740b8410 con 0x7f9674143d10 2024-03-27T03:11:35.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.948+0000 7f967911c640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:35.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.948+0000 7f967911c640 10 monclient(hunting): authenticate will time out at 1656.255071s 2024-03-27T03:11:35.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.952+0000 7f967359e640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f9674154260 0x7f96741e60b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:60053/0 (socket says 172.21.15.153:60053) 2024-03-27T03:11:35.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.952+0000 7f967359e640 1 -- 172.21.15.153:0/2476662970 learned_addr learned my addr 172.21.15.153:0/2476662970 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:35.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.952+0000 7f965b7fe640 10 client.?.objecter ms_handle_connect 0x7f9674154260 2024-03-27T03:11:35.948 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.952+0000 7f965b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:35.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.952+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323710745 0 0) 0x7f96740b64f0 con 0x7f9674154260 2024-03-27T03:11:35.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.952+0000 7f965b7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:11:35.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.952+0000 7f965b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:35.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.952+0000 7f965b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.952+0000 7f965b7fe640 10 monclient(hunting): my global_id is 4456 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.952+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9660004eb0 con 0x7f9674154260 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.953+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3893982426 0 0) 0x7f9660004eb0 con 0x7f9674154260 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.953+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96740b64f0 con 0x7f9674154260 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.953+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f966c004d00 con 0x7f9674154260 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.953+0000 7f965b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.953+0000 7f965b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.953+0000 7f965b7fe640 10 monclient(hunting): dump: 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:35.953 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.953+0000 7f965b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2586937310 0 0) 0x7f96740b64f0 con 0x7f9674154260 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 >> v1:172.21.15.153:6790/0 conn(0x7f9674143d10 legacy=0x7f9674153b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 >> v1:172.21.15.153:6789/0 conn(0x7f96740b1f60 legacy=0x7f96740b7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f965b7fe640 10 monclient: _finish_hunting 0 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f965b7fe640 1 monclient: found mon.b 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f965b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f965b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96740b86f0 con 0x7f9674154260 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f965b7fe640 10 monclient: _finish_auth 0 2024-03-27T03:11:35.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f965b7fe640 10 monclient: _check_auth_tickets 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f965b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f967911c640 5 monclient: authenticate success, global_id 4456 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f967911c640 10 monclient: _renew_subs 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f967911c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f967911c640 1 -- 172.21.15.153:0/2476662970 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96741546f0 con 0x7f9674154260 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.956+0000 7f967911c640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.957+0000 7f967911c640 10 monclient: _renew_subs 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.957+0000 7f967911c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.957+0000 7f967911c640 1 -- 172.21.15.153:0/2476662970 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96741e77d0 con 0x7f9674154260 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.957+0000 7f967911c640 1 librados: init done 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.957+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f966c001ed0 con 0x7f9674154260 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.957+0000 7f965b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.957+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f966c005330 con 0x7f9674154260 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.957+0000 7f965b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.957+0000 7f965b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:11:35.955 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.957+0000 7f965b7fe640 10 monclient: dump: 2024-03-27T03:11:35.956 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:35.956 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:35.956 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:35.956 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:35.956 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:35.956 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:35.956 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:35.956 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:35.956 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:35.956 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:35.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.957+0000 7f967159a640 4 set_mon_vals no callback set 2024-03-27T03:11:35.956 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.958+0000 7f967911c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:35.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.959+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f966c005490 con 0x7f9674154260 2024-03-27T03:11:35.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.959+0000 7f967911c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:35.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.959+0000 7f967911c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:35.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.959+0000 7f967911c640 1 -- 172.21.15.153:0/2476662970 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9654005080 con 0x7f9674154260 2024-03-27T03:11:35.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.960+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f966c085120 con 0x7f9674154260 2024-03-27T03:11:35.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.960+0000 7f965b7fe640 10 client.4456.objecter ms_dispatch 0x7f9674000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:11:35.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.960+0000 7f965b7fe640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:11:35.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.960+0000 7f965b7fe640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:11:35.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.960+0000 7f965b7fe640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-27T03:11:35.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.960+0000 7f9672d9d640 10 monclient: get_auth_request con 0x7f966006dc80 auth_method 0 2024-03-27T03:11:35.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.960+0000 7f965b7fe640 10 client.4456.objecter ms_handle_connect 0x7f966006dc80 2024-03-27T03:11:35.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.968+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f966c050a00 con 0x7f9674154260 2024-03-27T03:11:35.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.968+0000 7f965b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:35.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:35.968+0000 7f965b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:11:36.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.289+0000 7f967911c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:36.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.289+0000 7f967911c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:36.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.289+0000 7f967911c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:36.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.289+0000 7f967911c640 1 -- 172.21.15.153:0/2476662970 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9654005330 con 0x7f9674154260 2024-03-27T03:11:36.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.295+0000 7f965b7fe640 1 -- 172.21.15.153:0/2476662970 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f966c052930 con 0x7f9674154260 2024-03-27T03:11:36.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.295+0000 7f965b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:36.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.295+0000 7f965b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:11:36.290 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:11:36.290 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:11:36.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.302+0000 7f967911c640 10 librados: watch_flush enter 2024-03-27T03:11:36.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.302+0000 7f967911c640 10 librados: watch_flush exit 2024-03-27T03:11:36.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.302+0000 7f967911c640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-27T03:11:36.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.302+0000 7f967911c640 10 client.4456.objecter successfully canceled tick 2024-03-27T03:11:36.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.302+0000 7f967911c640 1 -- 172.21.15.153:0/2476662970 >> v1:172.21.15.153:6816/2637937781 conn(0x7f966006dc80 legacy=0x7f9660070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:36.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.302+0000 7f967911c640 10 monclient: shutdown 2024-03-27T03:11:36.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.302+0000 7f967911c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:36.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.302+0000 7f967911c640 1 -- 172.21.15.153:0/2476662970 >> v1:172.21.15.196:6789/0 conn(0x7f9674154260 legacy=0x7f96741e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:36.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.302+0000 7f967911c640 1 -- 172.21.15.153:0/2476662970 shutdown_connections 2024-03-27T03:11:36.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.302+0000 7f967911c640 1 -- 172.21.15.153:0/2476662970 >> 172.21.15.153:0/2476662970 conn(0x7f9674066a00 msgr2=0x7f9674144370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:36.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.303+0000 7f967911c640 1 -- 172.21.15.153:0/2476662970 shutdown_connections 2024-03-27T03:11:36.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.303+0000 7f967911c640 1 -- 172.21.15.153:0/2476662970 wait complete. 2024-03-27T03:11:36.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:36.303+0000 7f967911c640 1 librados: shutdown 2024-03-27T03:11:36.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:11:43.125 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:11:43.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.282+0000 7f9caf59e640 10 monclient: get_monmap_and_config 2024-03-27T03:11:43.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.282+0000 7f9caf59e640 10 monclient: build_initial_monmap 2024-03-27T03:11:43.276 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.282+0000 7f9caf59e640 10 monclient: monmap: 2024-03-27T03:11:43.276 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:43.276 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:43.276 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:11:43.283492+0000 2024-03-27T03:11:43.276 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:11:43.283492+0000 2024-03-27T03:11:43.276 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:43.276 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:43.276 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:43.276 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:11:43.276 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:11:43.276 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:43.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.283+0000 7f9caf59e640 1 Processor -- start 2024-03-27T03:11:43.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.284+0000 7f9caf59e640 1 -- start start 2024-03-27T03:11:43.277 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.284+0000 7f9caf59e640 10 monclient: init 2024-03-27T03:11:43.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.285+0000 7f9caf59e640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:43.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.285+0000 7f9caf59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:11:43.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.285+0000 7f9caf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f9cb00b2800 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.285+0000 7f9caf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f9cb00b3220 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.285+0000 7f9caf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f9cb01439c0 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.285+0000 7f9caf59e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb00b3840 con 0x7f9cb01439c0 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.285+0000 7f9caf59e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb006f230 con 0x7f9cb00b3220 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.285+0000 7f9caf59e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb006ef50 con 0x7f9cb00b2800 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.285+0000 7f9caf59e640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.285+0000 7f9caf59e640 10 monclient(hunting): authenticate will time out at 1663.590765s 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.290+0000 7f9caed9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f9cb01439c0 0x7f9cb0151c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60369/0 (socket says 172.21.15.153:60369) 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.290+0000 7f9caed9d640 1 -- 172.21.15.153:0/453380292 learned_addr learned my addr 172.21.15.153:0/453380292 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.290+0000 7f9cad59a640 1 -- 172.21.15.153:0/453380292 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191508087 0 0) 0x7f9cb00b3840 con 0x7f9cb01439c0 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.290+0000 7f9cad59a640 10 monclient(hunting): _negotiate 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.290+0000 7f9cad59a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.290+0000 7f9cad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.290+0000 7f9cad59a640 10 monclient(hunting): my global_id is 4425 2024-03-27T03:11:43.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.290+0000 7f9cad59a640 1 -- 172.21.15.153:0/453380292 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c98003520 con 0x7f9cb01439c0 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 1 -- 172.21.15.153:0/453380292 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 37437152 0 0) 0x7f9c98003520 con 0x7f9cb01439c0 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 1 -- 172.21.15.153:0/453380292 >> v1:172.21.15.153:6790/0 conn(0x7f9cb00b2800 legacy=0x7f9cb01432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 1 -- 172.21.15.153:0/453380292 >> v1:172.21.15.196:6789/0 conn(0x7f9cb00b3220 legacy=0x7f9cb0146710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 10 monclient: _finish_hunting 0 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 1 monclient: found mon.noname-a 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 1 -- 172.21.15.153:0/453380292 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb0071d30 con 0x7f9cb01439c0 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 10 monclient: _finish_auth 0 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 10 monclient: _check_auth_tickets 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 1 -- 172.21.15.153:0/453380292 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9ca8001a10 con 0x7f9cb01439c0 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:11:43.288 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 10 monclient: dump: 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9caf59e640 5 monclient: authenticate success, global_id 4425 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9caf59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 1 -- 172.21.15.153:0/453380292 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ca8001be0 con 0x7f9cb01439c0 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 1 -- 172.21.15.153:0/453380292 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9ca8002030 con 0x7f9cb01439c0 2024-03-27T03:11:43.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9cad59a640 10 monclient: dump: 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9caf59e640 10 monclient: get_monmap_and_config success 2024-03-27T03:11:43.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.291+0000 7f9caf59e640 4 set_mon_vals no callback set 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.292+0000 7f9caf59e640 10 monclient: shutdown 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.292+0000 7f9caf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.292+0000 7f9caf59e640 1 -- 172.21.15.153:0/453380292 >> v1:172.21.15.153:6789/0 conn(0x7f9cb01439c0 legacy=0x7f9cb0151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.295+0000 7f9caf59e640 1 -- 172.21.15.153:0/453380292 shutdown_connections 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.295+0000 7f9caf59e640 1 -- 172.21.15.153:0/453380292 >> 172.21.15.153:0/453380292 conn(0x7f9cb0066a00 msgr2=0x7f9cb0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.295+0000 7f9caf59e640 1 -- 172.21.15.153:0/453380292 shutdown_connections 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.295+0000 7f9caf59e640 1 -- 172.21.15.153:0/453380292 wait complete. 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.296+0000 7f9caf59e640 10 monclient: build_initial_monmap 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.296+0000 7f9caf59e640 10 monclient: monmap: 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:11:43.292 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.297+0000 7f9caf59e640 1 librados: starting msgr at 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.297+0000 7f9caf59e640 1 librados: starting objecter 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.297+0000 7f9caf59e640 1 Processor -- start 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.297+0000 7f9caf59e640 1 -- start start 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.297+0000 7f9caf59e640 1 librados: setting wanted keys 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.297+0000 7f9caf59e640 1 librados: calling monclient init 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.297+0000 7f9caf59e640 10 monclient: init 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.298+0000 7f9caf59e640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.298+0000 7f9caf59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.298+0000 7f9caf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f9cb00b2800 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:43.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.298+0000 7f9caf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f9cb00b3220 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:43.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.298+0000 7f9caf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f9cb01439c0 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:43.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.298+0000 7f9caf59e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb00aaf00 con 0x7f9cb00b2800 2024-03-27T03:11:43.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.298+0000 7f9caf59e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb01e3c00 con 0x7f9cb01439c0 2024-03-27T03:11:43.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.298+0000 7f9caf59e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb01e4ad0 con 0x7f9cb00b3220 2024-03-27T03:11:43.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.298+0000 7f9caf59e640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:43.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.298+0000 7f9caf59e640 10 monclient(hunting): authenticate will time out at 1663.603980s 2024-03-27T03:11:43.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.303+0000 7f9cae59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f9cb00b2800 0x7f9cb00a9750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60457/0 (socket says 172.21.15.153:60457) 2024-03-27T03:11:43.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.303+0000 7f9cae59c640 1 -- 172.21.15.153:0/2367230508 learned_addr learned my addr 172.21.15.153:0/2367230508 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:43.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.303+0000 7f9c9effd640 10 client.?.objecter ms_handle_connect 0x7f9cb00b2800 2024-03-27T03:11:43.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.303+0000 7f9c9effd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:43.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003199219 0 0) 0x7f9cb00aaf00 con 0x7f9cb00b2800 2024-03-27T03:11:43.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 10 monclient(hunting): _negotiate 2024-03-27T03:11:43.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:43.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:43.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 10 monclient(hunting): my global_id is 4428 2024-03-27T03:11:43.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c90004eb0 con 0x7f9cb00b2800 2024-03-27T03:11:43.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 10 client.?.objecter ms_handle_connect 0x7f9cb00b3220 2024-03-27T03:11:43.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:43.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009811563 0 0) 0x7f9cb01e4ad0 con 0x7f9cb00b3220 2024-03-27T03:11:43.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 10 monclient(hunting): _negotiate 2024-03-27T03:11:43.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:43.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:43.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 10 monclient(hunting): my global_id is 4478 2024-03-27T03:11:43.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.304+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cb00aaf00 con 0x7f9cb00b3220 2024-03-27T03:11:43.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 10 client.?.objecter ms_handle_connect 0x7f9cb01439c0 2024-03-27T03:11:43.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:43.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3398343014 0 0) 0x7f9cb00aaf00 con 0x7f9cb00b3220 2024-03-27T03:11:43.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cb01e4ad0 con 0x7f9cb00b3220 2024-03-27T03:11:43.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3745551777 0 0) 0x7f9c90004eb0 con 0x7f9cb00b2800 2024-03-27T03:11:43.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cb00aaf00 con 0x7f9cb00b2800 2024-03-27T03:11:43.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9ca0001830 con 0x7f9cb00b3220 2024-03-27T03:11:43.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:43.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:11:43.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 10 monclient(hunting): dump: 2024-03-27T03:11:43.304 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:43.304 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:43.304 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:43.304 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9ca40018e0 con 0x7f9cb00b2800 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.305+0000 7f9c9effd640 10 monclient(hunting): dump: 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:43.305 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 398376511 0 0) 0x7f9cb01e4ad0 con 0x7f9cb00b3220 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 >> v1:172.21.15.196:6789/0 conn(0x7f9cb01439c0 legacy=0x7f9cb01ee640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 >> v1:172.21.15.153:6789/0 conn(0x7f9cb00b2800 legacy=0x7f9cb00a9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9c9effd640 10 monclient: _finish_hunting 0 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9c9effd640 1 monclient: found mon.c 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9c9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9c9effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:11:43.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb01e3ea0 con 0x7f9cb00b3220 2024-03-27T03:11:43.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9c9effd640 10 monclient: _finish_auth 0 2024-03-27T03:11:43.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9c9effd640 10 monclient: _check_auth_tickets 2024-03-27T03:11:43.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9c9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:43.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9caf59e640 5 monclient: authenticate success, global_id 4478 2024-03-27T03:11:43.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9caf59e640 10 monclient: _renew_subs 2024-03-27T03:11:43.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9caf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:11:43.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9caf59e640 1 -- 172.21.15.153:0/2367230508 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb0149c50 con 0x7f9cb00b3220 2024-03-27T03:11:43.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9caf59e640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:11:43.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9caf59e640 10 monclient: _renew_subs 2024-03-27T03:11:43.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9caf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:11:43.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9caf59e640 1 -- 172.21.15.153:0/2367230508 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb0147410 con 0x7f9cb00b3220 2024-03-27T03:11:43.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.309+0000 7f9caf59e640 1 librados: init done 2024-03-27T03:11:43.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.310+0000 7f9caf59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:43.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.310+0000 7f9caf59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:43.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.310+0000 7f9caf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:11:43.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.310+0000 7f9caf59e640 1 -- 172.21.15.153:0/2367230508 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb004f730 con 0x7f9cb00b3220 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.311+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ca0001d40 con 0x7f9cb00b3220 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.311+0000 7f9c9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.311+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9ca0003830 con 0x7f9cb00b3220 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.311+0000 7f9c9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.311+0000 7f9c9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.311+0000 7f9c9effd640 10 monclient: dump: 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:43.310 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:43.311 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:43.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.312+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f9ca0019800 con 0x7f9cb00b3220 2024-03-27T03:11:43.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.312+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f9ca00836d0 con 0x7f9cb00b3220 2024-03-27T03:11:43.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.312+0000 7f9c9effd640 10 client.4478.objecter ms_dispatch 0x7f9cb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:11:43.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.312+0000 7f9c9effd640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:11:43.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.312+0000 7f9c9effd640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:11:43.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.313+0000 7f9c9effd640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-27T03:11:43.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.314+0000 7f9c9ffff640 4 set_mon_vals no callback set 2024-03-27T03:11:43.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.316+0000 7f9cae59c640 10 monclient: get_auth_request con 0x7f9c9006dee0 auth_method 0 2024-03-27T03:11:43.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.317+0000 7f9c9effd640 10 client.4478.objecter ms_handle_connect 0x7f9c9006dee0 2024-03-27T03:11:43.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.322+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ca004ede0 con 0x7f9cb00b3220 2024-03-27T03:11:43.315 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.322+0000 7f9c9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:43.316 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.322+0000 7f9c9effd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:11:43.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.655+0000 7f9caf59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:43.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.655+0000 7f9caf59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:43.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.655+0000 7f9caf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:11:43.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.655+0000 7f9caf59e640 1 -- 172.21.15.153:0/2367230508 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cb0147710 con 0x7f9cb00b3220 2024-03-27T03:11:43.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.656+0000 7f9c9effd640 1 -- 172.21.15.153:0/2367230508 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f9ca0050ee0 con 0x7f9cb00b3220 2024-03-27T03:11:43.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.656+0000 7f9c9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:43.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.656+0000 7f9c9effd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:11:43.650 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:11:43.650 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:11:43.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.660+0000 7f9caf59e640 10 librados: watch_flush enter 2024-03-27T03:11:43.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.660+0000 7f9caf59e640 10 librados: watch_flush exit 2024-03-27T03:11:43.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.660+0000 7f9caf59e640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-27T03:11:43.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.660+0000 7f9caf59e640 10 client.4478.objecter successfully canceled tick 2024-03-27T03:11:43.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.660+0000 7f9caf59e640 1 -- 172.21.15.153:0/2367230508 >> v1:172.21.15.153:6816/2637937781 conn(0x7f9c9006dee0 legacy=0x7f9c900703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:43.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.660+0000 7f9caf59e640 10 monclient: shutdown 2024-03-27T03:11:43.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.660+0000 7f9caf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:43.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.660+0000 7f9caf59e640 1 -- 172.21.15.153:0/2367230508 >> v1:172.21.15.153:6790/0 conn(0x7f9cb00b3220 legacy=0x7f9cb01e43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:43.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.665+0000 7f9caf59e640 1 -- 172.21.15.153:0/2367230508 shutdown_connections 2024-03-27T03:11:43.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.665+0000 7f9caf59e640 1 -- 172.21.15.153:0/2367230508 >> 172.21.15.153:0/2367230508 conn(0x7f9cb0066a00 msgr2=0x7f9cb00b7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:43.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.665+0000 7f9caf59e640 1 -- 172.21.15.153:0/2367230508 shutdown_connections 2024-03-27T03:11:43.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.665+0000 7f9caf59e640 1 -- 172.21.15.153:0/2367230508 wait complete. 2024-03-27T03:11:43.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:43.665+0000 7f9caf59e640 1 librados: shutdown 2024-03-27T03:11:43.676 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:11:50.483 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:11:50.633 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.640+0000 7f49395c8640 10 monclient: get_monmap_and_config 2024-03-27T03:11:50.633 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.640+0000 7f49395c8640 10 monclient: build_initial_monmap 2024-03-27T03:11:50.633 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.640+0000 7f49395c8640 10 monclient: monmap: 2024-03-27T03:11:50.633 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:50.633 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:50.633 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:11:50.641176+0000 2024-03-27T03:11:50.633 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:11:50.641176+0000 2024-03-27T03:11:50.634 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:50.634 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:50.634 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:50.634 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:11:50.634 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:11:50.634 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.641+0000 7f49395c8640 1 Processor -- start 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.641+0000 7f49395c8640 1 -- start start 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.641+0000 7f49395c8640 10 monclient: init 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.642+0000 7f49395c8640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.642+0000 7f49395c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.642+0000 7f49395c8640 10 monclient(hunting): picked mon.noname-b con 0x7f49340b1f60 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.642+0000 7f49395c8640 10 monclient(hunting): picked mon.noname-c con 0x7f4934071eb0 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.642+0000 7f49395c8640 10 monclient(hunting): picked mon.noname-a con 0x7f4934143d10 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.642+0000 7f49395c8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493406ef80 con 0x7f4934143d10 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.642+0000 7f49395c8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493406f260 con 0x7f4934071eb0 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.642+0000 7f49395c8640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49340b40c0 con 0x7f49340b1f60 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.642+0000 7f49395c8640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:50.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.642+0000 7f49395c8640 10 monclient(hunting): authenticate will time out at 1670.947774s 2024-03-27T03:11:50.636 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.643+0000 7f4932ffd640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f49340b1f60 0x7f4934143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:49778/0 (socket says 172.21.15.153:49778) 2024-03-27T03:11:50.637 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.643+0000 7f4932ffd640 1 -- 172.21.15.153:0/1529173222 learned_addr learned my addr 172.21.15.153:0/1529173222 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:50.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.644+0000 7f4931ffb640 1 -- 172.21.15.153:0/1529173222 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737022402 0 0) 0x7f49340b40c0 con 0x7f49340b1f60 2024-03-27T03:11:50.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.644+0000 7f4931ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:11:50.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.644+0000 7f4931ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:50.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.644+0000 7f4931ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:50.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.644+0000 7f4931ffb640 10 monclient(hunting): my global_id is 4481 2024-03-27T03:11:50.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.644+0000 7f4931ffb640 1 -- 172.21.15.153:0/1529173222 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4920003520 con 0x7f49340b1f60 2024-03-27T03:11:50.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.644+0000 7f4931ffb640 1 -- 172.21.15.153:0/1529173222 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3619771243 0 0) 0x7f4920003520 con 0x7f49340b1f60 2024-03-27T03:11:50.638 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.644+0000 7f4931ffb640 1 -- 172.21.15.153:0/1529173222 >> v1:172.21.15.196:6789/0 conn(0x7f4934071eb0 legacy=0x7f4934146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:11:50.639 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.645+0000 7f4931ffb640 1 -- 172.21.15.153:0/1529173222 >> v1:172.21.15.153:6789/0 conn(0x7f4934143d10 legacy=0x7f4934146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:11:50.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f4931ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:11:50.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f4931ffb640 1 monclient: found mon.noname-b 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f4931ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f4931ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f4931ffb640 1 -- 172.21.15.153:0/1529173222 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4934155d70 con 0x7f49340b1f60 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f4931ffb640 10 monclient: _finish_auth 0 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f4931ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f4931ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f4931ffb640 1 -- 172.21.15.153:0/1529173222 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4924001690 con 0x7f49340b1f60 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f4931ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f4931ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f4931ffb640 10 monclient: dump: 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:50.643 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:50.644 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:50.644 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:50.644 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:50.644 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:50.644 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:50.644 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:50.644 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:50.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f49395c8640 5 monclient: authenticate success, global_id 4481 2024-03-27T03:11:50.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.649+0000 7f49395c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:11:50.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.651+0000 7f4931ffb640 1 -- 172.21.15.153:0/1529173222 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49240018f0 con 0x7f49340b1f60 2024-03-27T03:11:50.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.651+0000 7f4931ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:50.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.651+0000 7f4931ffb640 1 -- 172.21.15.153:0/1529173222 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4924001d40 con 0x7f49340b1f60 2024-03-27T03:11:50.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.651+0000 7f4931ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.651+0000 7f4931ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.651+0000 7f4931ffb640 10 monclient: dump: 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.651+0000 7f49395c8640 10 monclient: get_monmap_and_config success 2024-03-27T03:11:50.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.651+0000 7f49395c8640 4 set_mon_vals no callback set 2024-03-27T03:11:50.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.652+0000 7f49395c8640 10 monclient: shutdown 2024-03-27T03:11:50.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.652+0000 7f49395c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:50.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.652+0000 7f49395c8640 1 -- 172.21.15.153:0/1529173222 >> v1:172.21.15.153:6790/0 conn(0x7f49340b1f60 legacy=0x7f4934143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:50.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.652+0000 7f49395c8640 1 -- 172.21.15.153:0/1529173222 shutdown_connections 2024-03-27T03:11:50.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.652+0000 7f49395c8640 1 -- 172.21.15.153:0/1529173222 >> 172.21.15.153:0/1529173222 conn(0x7f4934066a00 msgr2=0x7f4934066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:50.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.652+0000 7f49395c8640 1 -- 172.21.15.153:0/1529173222 shutdown_connections 2024-03-27T03:11:50.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.652+0000 7f49395c8640 1 -- 172.21.15.153:0/1529173222 wait complete. 2024-03-27T03:11:50.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.653+0000 7f49395c8640 10 monclient: build_initial_monmap 2024-03-27T03:11:50.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.653+0000 7f49395c8640 10 monclient: monmap: 2024-03-27T03:11:50.649 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:50.649 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:11:50.649 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:11:50.649 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:11:50.649 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:50.649 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:50.649 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:50.649 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:11:50.649 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:11:50.649 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:50.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.653+0000 7f49395c8640 1 librados: starting msgr at 2024-03-27T03:11:50.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.653+0000 7f49395c8640 1 librados: starting objecter 2024-03-27T03:11:50.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.654+0000 7f49395c8640 1 Processor -- start 2024-03-27T03:11:50.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.654+0000 7f49395c8640 1 -- start start 2024-03-27T03:11:50.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.654+0000 7f49395c8640 1 librados: setting wanted keys 2024-03-27T03:11:50.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.654+0000 7f49395c8640 1 librados: calling monclient init 2024-03-27T03:11:50.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.654+0000 7f49395c8640 10 monclient: init 2024-03-27T03:11:50.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.655+0000 7f49395c8640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:50.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.655+0000 7f49395c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:11:50.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.658+0000 7f49395c8640 10 monclient(hunting): picked mon.noname-b con 0x7f49340b1f60 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:50.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.658+0000 7f49395c8640 10 monclient(hunting): picked mon.noname-a con 0x7f4934143d10 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:50.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.658+0000 7f49395c8640 10 monclient(hunting): picked mon.noname-c con 0x7f49340ab8a0 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:50.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.658+0000 7f49395c8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49340ab490 con 0x7f4934143d10 2024-03-27T03:11:50.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.658+0000 7f49395c8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49340aca70 con 0x7f49340b1f60 2024-03-27T03:11:50.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.659+0000 7f49395c8640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49340acd50 con 0x7f49340ab8a0 2024-03-27T03:11:50.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.659+0000 7f49395c8640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:50.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.659+0000 7f49395c8640 10 monclient(hunting): authenticate will time out at 1670.964596s 2024-03-27T03:11:50.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.660+0000 7f49337fe640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f49340ab8a0 0x7f49340a9cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:40024/0 (socket says 172.21.15.153:40024) 2024-03-27T03:11:50.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.660+0000 7f49337fe640 1 -- 172.21.15.153:0/1267265393 learned_addr learned my addr 172.21.15.153:0/1267265393 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:50.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.663+0000 7f492b7fe640 10 client.?.objecter ms_handle_connect 0x7f49340b1f60 2024-03-27T03:11:50.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.663+0000 7f492b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:50.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.663+0000 7f492b7fe640 10 client.?.objecter ms_handle_connect 0x7f49340ab8a0 2024-03-27T03:11:50.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.663+0000 7f492b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:50.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.663+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245104785 0 0) 0x7f49340aca70 con 0x7f49340b1f60 2024-03-27T03:11:50.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.663+0000 7f492b7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:11:50.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.663+0000 7f492b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:50.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.663+0000 7f492b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:50.657 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.663+0000 7f492b7fe640 10 monclient(hunting): my global_id is 4462 2024-03-27T03:11:50.657 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.663+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4914004eb0 con 0x7f49340b1f60 2024-03-27T03:11:50.657 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.663+0000 7f492b7fe640 10 client.?.objecter ms_handle_connect 0x7f4934143d10 2024-03-27T03:11:50.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.663+0000 7f492b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:50.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175607615 0 0) 0x7f49340ab490 con 0x7f4934143d10 2024-03-27T03:11:50.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:11:50.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:50.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:50.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 10 monclient(hunting): my global_id is 4431 2024-03-27T03:11:50.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49340aca70 con 0x7f4934143d10 2024-03-27T03:11:50.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138702476 0 0) 0x7f49340acd50 con 0x7f49340ab8a0 2024-03-27T03:11:50.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:11:50.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 10 monclient(hunting): my global_id is 4484 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49340ab490 con 0x7f49340ab8a0 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 190020290 0 0) 0x7f49340aca70 con 0x7f4934143d10 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49340acd50 con 0x7f4934143d10 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f491c001810 con 0x7f4934143d10 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 10 monclient(hunting): dump: 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:50.664 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.664+0000 7f492b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4286617717 0 0) 0x7f49340acd50 con 0x7f4934143d10 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 >> v1:172.21.15.153:6790/0 conn(0x7f49340ab8a0 legacy=0x7f49340a9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 >> v1:172.21.15.196:6789/0 conn(0x7f49340b1f60 legacy=0x7f4934070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f492b7fe640 10 monclient: _finish_hunting 0 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f492b7fe640 1 monclient: found mon.a 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f492b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f492b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4934152a00 con 0x7f4934143d10 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f492b7fe640 10 monclient: _finish_auth 0 2024-03-27T03:11:50.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f492b7fe640 10 monclient: _check_auth_tickets 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f492b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f49395c8640 5 monclient: authenticate success, global_id 4431 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f49395c8640 10 monclient: _renew_subs 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f49395c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f49395c8640 1 -- 172.21.15.153:0/1267265393 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4934200c50 con 0x7f4934143d10 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f49395c8640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f49395c8640 10 monclient: _renew_subs 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f49395c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f49395c8640 1 -- 172.21.15.153:0/1267265393 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4934200fc0 con 0x7f4934143d10 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.665+0000 7f49395c8640 1 librados: init done 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.666+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f491c0019f0 con 0x7f4934143d10 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.666+0000 7f492b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.666+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f491c003830 con 0x7f4934143d10 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.666+0000 7f492b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:50.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.666+0000 7f492b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.666+0000 7f492b7fe640 10 monclient: dump: 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.666+0000 7f49317fa640 4 set_mon_vals no callback set 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.667+0000 7f49395c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:50.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.667+0000 7f49395c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:50.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.667+0000 7f49395c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:50.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.667+0000 7f49395c8640 1 -- 172.21.15.153:0/1267265393 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4918005080 con 0x7f4934143d10 2024-03-27T03:11:50.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.669+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f491c019800 con 0x7f4934143d10 2024-03-27T03:11:50.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.669+0000 7f4932ffd640 10 monclient: get_auth_request con 0x7f491406de50 auth_method 0 2024-03-27T03:11:50.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.670+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f491c083870 con 0x7f4934143d10 2024-03-27T03:11:50.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.670+0000 7f492b7fe640 10 client.4431.objecter ms_dispatch 0x7f4934000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:11:50.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.670+0000 7f492b7fe640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:11:50.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.670+0000 7f492b7fe640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:11:50.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.670+0000 7f492b7fe640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-27T03:11:50.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.670+0000 7f492b7fe640 10 client.4431.objecter ms_handle_connect 0x7f491406de50 2024-03-27T03:11:50.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.679+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f491c04ef80 con 0x7f4934143d10 2024-03-27T03:11:50.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.679+0000 7f492b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:50.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:50.679+0000 7f492b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:11:51.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.016+0000 7f49395c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:51.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.016+0000 7f49395c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:51.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.016+0000 7f49395c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:11:51.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.016+0000 7f49395c8640 1 -- 172.21.15.153:0/1267265393 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4918005330 con 0x7f4934143d10 2024-03-27T03:11:51.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.017+0000 7f492b7fe640 1 -- 172.21.15.153:0/1267265393 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f491c051080 con 0x7f4934143d10 2024-03-27T03:11:51.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.017+0000 7f492b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:51.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.017+0000 7f492b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:11:51.010 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:11:51.010 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:11:51.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.021+0000 7f49395c8640 10 librados: watch_flush enter 2024-03-27T03:11:51.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.021+0000 7f49395c8640 10 librados: watch_flush exit 2024-03-27T03:11:51.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.021+0000 7f49395c8640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-27T03:11:51.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.021+0000 7f49395c8640 10 client.4431.objecter successfully canceled tick 2024-03-27T03:11:51.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.022+0000 7f49395c8640 1 -- 172.21.15.153:0/1267265393 >> v1:172.21.15.153:6816/2637937781 conn(0x7f491406de50 legacy=0x7f4914070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:51.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.022+0000 7f49395c8640 10 monclient: shutdown 2024-03-27T03:11:51.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.022+0000 7f49395c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:51.015 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.022+0000 7f49395c8640 1 -- 172.21.15.153:0/1267265393 >> v1:172.21.15.153:6789/0 conn(0x7f4934143d10 legacy=0x7f49340a95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:51.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.022+0000 7f49395c8640 1 -- 172.21.15.153:0/1267265393 shutdown_connections 2024-03-27T03:11:51.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.022+0000 7f49395c8640 1 -- 172.21.15.153:0/1267265393 >> 172.21.15.153:0/1267265393 conn(0x7f4934066a00 msgr2=0x7f49340b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:51.016 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.024+0000 7f49395c8640 1 -- 172.21.15.153:0/1267265393 shutdown_connections 2024-03-27T03:11:51.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.024+0000 7f49395c8640 1 -- 172.21.15.153:0/1267265393 wait complete. 2024-03-27T03:11:51.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:51.025+0000 7f49395c8640 1 librados: shutdown 2024-03-27T03:11:51.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:11:57.843 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:11:57.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.996+0000 7f3a1264e640 10 monclient: get_monmap_and_config 2024-03-27T03:11:57.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.996+0000 7f3a1264e640 10 monclient: build_initial_monmap 2024-03-27T03:11:57.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.996+0000 7f3a1264e640 10 monclient: monmap: 2024-03-27T03:11:57.989 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:57.990 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:57.990 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:11:57.997367+0000 2024-03-27T03:11:57.990 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:11:57.997367+0000 2024-03-27T03:11:57.990 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:57.990 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:57.990 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:57.990 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:11:57.990 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:11:57.990 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:57.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.997+0000 7f3a1264e640 1 Processor -- start 2024-03-27T03:11:57.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.997+0000 7f3a1264e640 1 -- start start 2024-03-27T03:11:57.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.997+0000 7f3a1264e640 10 monclient: init 2024-03-27T03:11:57.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.998+0000 7f3a1264e640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:57.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.998+0000 7f3a1264e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:11:57.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.998+0000 7f3a1264e640 10 monclient(hunting): picked mon.noname-c con 0x7f3a0c1422b0 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:57.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.998+0000 7f3a1264e640 10 monclient(hunting): picked mon.noname-a con 0x7f3a0c142690 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:57.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.998+0000 7f3a1264e640 10 monclient(hunting): picked mon.noname-b con 0x7f3a0c0b17d0 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:57.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.998+0000 7f3a1264e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c142c80 con 0x7f3a0c142690 2024-03-27T03:11:57.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.998+0000 7f3a1264e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c0aae40 con 0x7f3a0c1422b0 2024-03-27T03:11:57.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.998+0000 7f3a1264e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c13fbd0 con 0x7f3a0c0b17d0 2024-03-27T03:11:57.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.998+0000 7f3a1264e640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:57.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.998+0000 7f3a1264e640 10 monclient(hunting): authenticate will time out at 1678.304573s 2024-03-27T03:11:57.993 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.999+0000 7f3a0bfff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3a0c1422b0 0x7f3a0c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:34844/0 (socket says 172.21.15.153:34844) 2024-03-27T03:11:57.993 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.999+0000 7f3a0bfff640 1 -- 172.21.15.153:0/3741421086 learned_addr learned my addr 172.21.15.153:0/3741421086 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:57.993 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.999+0000 7f3a0affd640 1 -- 172.21.15.153:0/3741421086 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774243554 0 0) 0x7f3a0c0aae40 con 0x7f3a0c1422b0 2024-03-27T03:11:57.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.999+0000 7f3a0affd640 10 monclient(hunting): _negotiate 2024-03-27T03:11:57.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.999+0000 7f3a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:57.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.999+0000 7f3a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:57.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.999+0000 7f3a0affd640 10 monclient(hunting): my global_id is 4465 2024-03-27T03:11:57.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:57.999+0000 7f3a0affd640 1 -- 172.21.15.153:0/3741421086 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f4003520 con 0x7f3a0c1422b0 2024-03-27T03:11:57.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 1 -- 172.21.15.153:0/3741421086 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2947712388 0 0) 0x7f39f4003520 con 0x7f3a0c1422b0 2024-03-27T03:11:57.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 1 -- 172.21.15.153:0/3741421086 >> v1:172.21.15.153:6790/0 conn(0x7f3a0c0b17d0 legacy=0x7f3a0c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:11:57.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 1 -- 172.21.15.153:0/3741421086 >> v1:172.21.15.153:6789/0 conn(0x7f3a0c142690 legacy=0x7f3a0c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:57.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 10 monclient: _finish_hunting 0 2024-03-27T03:11:57.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 1 monclient: found mon.noname-c 2024-03-27T03:11:57.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 1 -- 172.21.15.153:0/3741421086 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c151940 con 0x7f3a0c1422b0 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 10 monclient: _finish_auth 0 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 10 monclient: _check_auth_tickets 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 1 -- 172.21.15.153:0/3741421086 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3a00001670 con 0x7f3a0c1422b0 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a0affd640 10 monclient: dump: 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:57.995 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:57.996 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:57.996 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:57.996 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:57.996 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:57.996 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:57.996 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a1264e640 5 monclient: authenticate success, global_id 4465 2024-03-27T03:11:57.996 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.000+0000 7f3a1264e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:11:57.998 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.001+0000 7f3a0affd640 1 -- 172.21.15.153:0/3741421086 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a000017d0 con 0x7f3a0c1422b0 2024-03-27T03:11:57.998 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.001+0000 7f3a0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:57.998 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.001+0000 7f3a0affd640 1 -- 172.21.15.153:0/3741421086 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3a00001be0 con 0x7f3a0c1422b0 2024-03-27T03:11:57.998 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.001+0000 7f3a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:57.998 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.001+0000 7f3a0affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:11:57.998 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.001+0000 7f3a0affd640 10 monclient: dump: 2024-03-27T03:11:57.998 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:57.998 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.001+0000 7f3a1264e640 10 monclient: get_monmap_and_config success 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.001+0000 7f3a1264e640 4 set_mon_vals no callback set 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.002+0000 7f3a1264e640 10 monclient: shutdown 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.002+0000 7f3a1264e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.002+0000 7f3a1264e640 1 -- 172.21.15.153:0/3741421086 >> v1:172.21.15.196:6789/0 conn(0x7f3a0c1422b0 legacy=0x7f3a0c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.002+0000 7f3a1264e640 1 -- 172.21.15.153:0/3741421086 shutdown_connections 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.002+0000 7f3a1264e640 1 -- 172.21.15.153:0/3741421086 >> 172.21.15.153:0/3741421086 conn(0x7f3a0c066a00 msgr2=0x7f3a0c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:57.999 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.002+0000 7f3a1264e640 1 -- 172.21.15.153:0/3741421086 shutdown_connections 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.002+0000 7f3a1264e640 1 -- 172.21.15.153:0/3741421086 wait complete. 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.003+0000 7f3a1264e640 10 monclient: build_initial_monmap 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.003+0000 7f3a1264e640 10 monclient: monmap: 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:58.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.004+0000 7f3a1264e640 1 librados: starting msgr at 2024-03-27T03:11:58.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.004+0000 7f3a1264e640 1 librados: starting objecter 2024-03-27T03:11:58.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.004+0000 7f3a1264e640 1 Processor -- start 2024-03-27T03:11:58.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.005+0000 7f3a1264e640 1 -- start start 2024-03-27T03:11:58.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.005+0000 7f3a1264e640 1 librados: setting wanted keys 2024-03-27T03:11:58.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.005+0000 7f3a1264e640 1 librados: calling monclient init 2024-03-27T03:11:58.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.005+0000 7f3a1264e640 10 monclient: init 2024-03-27T03:11:58.006 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a1264e640 10 monclient: _reopen_session rank -1 2024-03-27T03:11:58.006 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a1264e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:11:58.006 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a1264e640 10 monclient(hunting): picked mon.noname-c con 0x7f3a0c0b17d0 addr v1:172.21.15.153:6790/0 2024-03-27T03:11:58.006 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a1264e640 10 monclient(hunting): picked mon.noname-b con 0x7f3a0c1422b0 addr v1:172.21.15.196:6789/0 2024-03-27T03:11:58.006 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a1264e640 10 monclient(hunting): picked mon.noname-a con 0x7f3a0c142690 addr v1:172.21.15.153:6789/0 2024-03-27T03:11:58.006 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a1264e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c0b0ca0 con 0x7f3a0c142690 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a1264e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c0b0f80 con 0x7f3a0c1422b0 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a0b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3a0c1422b0 0x7f3a0c0acf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:35796/0 (socket says 172.21.15.153:35796) 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a0b7fe640 1 -- 172.21.15.153:0/986945265 learned_addr learned my addr 172.21.15.153:0/986945265 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a1264e640 1 -- 172.21.15.153:0/986945265 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c0b1260 con 0x7f3a0c0b17d0 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a1264e640 10 monclient(hunting): _renew_subs 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a1264e640 10 monclient(hunting): authenticate will time out at 1678.312331s 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a08ff9640 10 client.?.objecter ms_handle_connect 0x7f3a0c1422b0 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a08ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a08ff9640 10 client.?.objecter ms_handle_connect 0x7f3a0c0b17d0 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.006+0000 7f3a08ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053459595 0 0) 0x7f3a0c0b0f80 con 0x7f3a0c1422b0 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:58.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 10 monclient(hunting): my global_id is 4468 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f8004eb0 con 0x7f3a0c1422b0 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2318222347 0 0) 0x7f39f8004eb0 con 0x7f3a0c1422b0 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a0c0b0f80 con 0x7f3a0c1422b0 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f39fc001990 con 0x7f3a0c1422b0 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 10 monclient(hunting): dump: 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:58.008 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.007+0000 7f3a08ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 10 client.?.objecter ms_handle_connect 0x7f3a0c142690 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602392571 0 0) 0x7f3a0c0b1260 con 0x7f3a0c0b17d0 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 10 monclient(hunting): my global_id is 4487 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f8004eb0 con 0x7f3a0c0b17d0 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1215880093 0 0) 0x7f3a0c0b0f80 con 0x7f3a0c1422b0 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 >> v1:172.21.15.153:6790/0 conn(0x7f3a0c0b17d0 legacy=0x7f3a0c0ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 >> v1:172.21.15.153:6789/0 conn(0x7f3a0c142690 legacy=0x7f3a0c1e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:58.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 1 monclient: found mon.b 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c0aff90 con 0x7f3a0c1422b0 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 10 monclient: _finish_auth 0 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a1264e640 5 monclient: authenticate success, global_id 4468 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a1264e640 10 monclient: _renew_subs 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a1264e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a1264e640 1 -- 172.21.15.153:0/986945265 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0c0afa80 con 0x7f3a0c1422b0 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a1264e640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:11:58.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a1264e640 10 monclient: _renew_subs 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a1264e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a1264e640 1 -- 172.21.15.153:0/986945265 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0c0af050 con 0x7f3a0c1422b0 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.008+0000 7f3a1264e640 1 librados: init done 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.009+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39fc001c50 con 0x7f3a0c1422b0 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.009+0000 7f3a08ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.009+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f39fc003a60 con 0x7f3a0c1422b0 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.009+0000 7f3a08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.009+0000 7f3a08ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.009+0000 7f3a08ff9640 10 monclient: dump: 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:11:58.011 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.009+0000 7f3a0a7fc640 4 set_mon_vals no callback set 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.010+0000 7f3a1264e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.010+0000 7f3a1264e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.010+0000 7f3a1264e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.011+0000 7f3a1264e640 1 -- 172.21.15.153:0/986945265 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ec005080 con 0x7f3a0c1422b0 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.011+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f39fc003bc0 con 0x7f3a0c1422b0 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.012+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f39fc084ad0 con 0x7f3a0c1422b0 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.012+0000 7f3a08ff9640 10 client.4468.objecter ms_dispatch 0x7f3a0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:11:58.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.012+0000 7f3a08ff9640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:11:58.013 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.012+0000 7f3a08ff9640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:11:58.013 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.012+0000 7f3a08ff9640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-27T03:11:58.013 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.012+0000 7f3a0bfff640 10 monclient: get_auth_request con 0x7f39f806de00 auth_method 0 2024-03-27T03:11:58.013 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.012+0000 7f3a08ff9640 10 client.4468.objecter ms_handle_connect 0x7f39f806de00 2024-03-27T03:11:58.013 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.018+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39fc0503b0 con 0x7f3a0c1422b0 2024-03-27T03:11:58.013 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.018+0000 7f3a08ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:11:58.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.018+0000 7f3a08ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:11:58.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.384+0000 7f3a1264e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:58.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.384+0000 7f3a1264e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:58.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.384+0000 7f3a1264e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:11:58.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.384+0000 7f3a1264e640 1 -- 172.21.15.153:0/986945265 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39ec005330 con 0x7f3a0c1422b0 2024-03-27T03:11:58.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.385+0000 7f3a08ff9640 1 -- 172.21.15.153:0/986945265 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f39fc0522e0 con 0x7f3a0c1422b0 2024-03-27T03:11:58.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.385+0000 7f3a08ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:11:58.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.385+0000 7f3a08ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:11:58.378 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:11:58.379 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:11:58.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.388+0000 7f3a1264e640 10 librados: watch_flush enter 2024-03-27T03:11:58.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.388+0000 7f3a1264e640 10 librados: watch_flush exit 2024-03-27T03:11:58.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.388+0000 7f3a1264e640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-27T03:11:58.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.388+0000 7f3a1264e640 10 client.4468.objecter successfully canceled tick 2024-03-27T03:11:58.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.389+0000 7f3a1264e640 1 -- 172.21.15.153:0/986945265 >> v1:172.21.15.153:6816/2637937781 conn(0x7f39f806de00 legacy=0x7f39f80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:58.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.389+0000 7f3a1264e640 10 monclient: shutdown 2024-03-27T03:11:58.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.389+0000 7f3a1264e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:11:58.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.389+0000 7f3a1264e640 1 -- 172.21.15.153:0/986945265 >> v1:172.21.15.196:6789/0 conn(0x7f3a0c1422b0 legacy=0x7f3a0c0acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:11:58.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.389+0000 7f3a1264e640 1 -- 172.21.15.153:0/986945265 shutdown_connections 2024-03-27T03:11:58.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.389+0000 7f3a1264e640 1 -- 172.21.15.153:0/986945265 >> 172.21.15.153:0/986945265 conn(0x7f3a0c066a00 msgr2=0x7f3a0c13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:11:58.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.389+0000 7f3a1264e640 1 -- 172.21.15.153:0/986945265 shutdown_connections 2024-03-27T03:11:58.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.389+0000 7f3a1264e640 1 -- 172.21.15.153:0/986945265 wait complete. 2024-03-27T03:11:58.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:11:58.389+0000 7f3a1264e640 1 librados: shutdown 2024-03-27T03:11:58.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:12:05.206 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:12:05.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.384+0000 7f43d950e640 10 monclient: get_monmap_and_config 2024-03-27T03:12:05.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.384+0000 7f43d950e640 10 monclient: build_initial_monmap 2024-03-27T03:12:05.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.384+0000 7f43d950e640 10 monclient: monmap: 2024-03-27T03:12:05.378 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:05.378 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:05.378 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:12:05.385785+0000 2024-03-27T03:12:05.378 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:12:05.385785+0000 2024-03-27T03:12:05.378 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:05.378 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:05.378 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:05.378 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:12:05.378 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:12:05.379 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:05.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.385+0000 7f43d950e640 1 Processor -- start 2024-03-27T03:12:05.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.385+0000 7f43d950e640 1 -- start start 2024-03-27T03:12:05.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.386+0000 7f43d950e640 10 monclient: init 2024-03-27T03:12:05.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.387+0000 7f43d950e640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:05.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.387+0000 7f43d950e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:12:05.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.387+0000 7f43d950e640 10 monclient(hunting): picked mon.noname-c con 0x7f43d4070170 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:05.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.387+0000 7f43d950e640 10 monclient(hunting): picked mon.noname-a con 0x7f43d40713f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:05.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.387+0000 7f43d950e640 10 monclient(hunting): picked mon.noname-b con 0x7f43d4070550 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:05.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.387+0000 7f43d950e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d413dbd0 con 0x7f43d40713f0 2024-03-27T03:12:05.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.387+0000 7f43d950e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d413ff70 con 0x7f43d4070170 2024-03-27T03:12:05.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.387+0000 7f43d950e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d4140250 con 0x7f43d4070550 2024-03-27T03:12:05.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.387+0000 7f43d950e640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:05.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.387+0000 7f43d950e640 10 monclient(hunting): authenticate will time out at 1685.693177s 2024-03-27T03:12:05.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d27fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f43d40713f0 0x7f43d41406e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:35290/0 (socket says 172.21.15.153:35290) 2024-03-27T03:12:05.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d27fc640 1 -- 172.21.15.153:0/775760222 learned_addr learned my addr 172.21.15.153:0/775760222 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:05.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d1ffb640 1 -- 172.21.15.153:0/775760222 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511945535 0 0) 0x7f43d413ff70 con 0x7f43d4070170 2024-03-27T03:12:05.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d1ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:12:05.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:05.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:05.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d1ffb640 10 monclient(hunting): my global_id is 4471 2024-03-27T03:12:05.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d1ffb640 1 -- 172.21.15.153:0/775760222 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43bc003520 con 0x7f43d4070170 2024-03-27T03:12:05.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d1ffb640 1 -- 172.21.15.153:0/775760222 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746883935 0 0) 0x7f43c4001110 con 0x7f43d40713f0 2024-03-27T03:12:05.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d1ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:12:05.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:05.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d1ffb640 10 monclient(hunting): my global_id is 4437 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.388+0000 7f43d1ffb640 1 -- 172.21.15.153:0/775760222 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d413ff70 con 0x7f43d40713f0 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 1 -- 172.21.15.153:0/775760222 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 944720795 0 0) 0x7f43bc003520 con 0x7f43d4070170 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 1 -- 172.21.15.153:0/775760222 >> v1:172.21.15.153:6790/0 conn(0x7f43d4070550 legacy=0x7f43d414b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 1 -- 172.21.15.153:0/775760222 >> v1:172.21.15.153:6789/0 conn(0x7f43d40713f0 legacy=0x7f43d41406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 1 monclient: found mon.noname-c 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 1 -- 172.21.15.153:0/775760222 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d4140df0 con 0x7f43d4070170 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 10 monclient: _finish_auth 0 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:12:05.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 1 -- 172.21.15.153:0/775760222 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f43c8001690 con 0x7f43d4070170 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d1ffb640 10 monclient: dump: 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d950e640 5 monclient: authenticate success, global_id 4471 2024-03-27T03:12:05.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.389+0000 7f43d950e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.390+0000 7f43d1ffb640 1 -- 172.21.15.153:0/775760222 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43c8001870 con 0x7f43d4070170 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.390+0000 7f43d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.390+0000 7f43d1ffb640 1 -- 172.21.15.153:0/775760222 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f43c8001c80 con 0x7f43d4070170 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.390+0000 7f43d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.390+0000 7f43d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.390+0000 7f43d1ffb640 10 monclient: dump: 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:05.388 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:05.389 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:05.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.390+0000 7f43d950e640 10 monclient: get_monmap_and_config success 2024-03-27T03:12:05.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.390+0000 7f43d950e640 4 set_mon_vals no callback set 2024-03-27T03:12:05.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.391+0000 7f43d950e640 10 monclient: shutdown 2024-03-27T03:12:05.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.391+0000 7f43d950e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:05.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.391+0000 7f43d950e640 1 -- 172.21.15.153:0/775760222 >> v1:172.21.15.196:6789/0 conn(0x7f43d4070170 legacy=0x7f43d413d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:05.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.391+0000 7f43d950e640 1 -- 172.21.15.153:0/775760222 shutdown_connections 2024-03-27T03:12:05.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.391+0000 7f43d950e640 1 -- 172.21.15.153:0/775760222 >> 172.21.15.153:0/775760222 conn(0x7f43d4067610 msgr2=0x7f43d40b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:05.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.392+0000 7f43d950e640 1 -- 172.21.15.153:0/775760222 shutdown_connections 2024-03-27T03:12:05.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.392+0000 7f43d950e640 1 -- 172.21.15.153:0/775760222 wait complete. 2024-03-27T03:12:05.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.393+0000 7f43d950e640 10 monclient: build_initial_monmap 2024-03-27T03:12:05.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.393+0000 7f43d950e640 10 monclient: monmap: 2024-03-27T03:12:05.394 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:05.394 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:12:05.394 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:12:05.394 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:12:05.394 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:05.394 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:05.394 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.393+0000 7f43d950e640 1 librados: starting msgr at 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.393+0000 7f43d950e640 1 librados: starting objecter 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.394+0000 7f43d950e640 1 Processor -- start 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.394+0000 7f43d950e640 1 -- start start 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.394+0000 7f43d950e640 1 librados: setting wanted keys 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.394+0000 7f43d950e640 1 librados: calling monclient init 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.394+0000 7f43d950e640 10 monclient: init 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43d950e640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43d950e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43d950e640 10 monclient(hunting): picked mon.noname-b con 0x7f43d4070170 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43d950e640 10 monclient(hunting): picked mon.noname-c con 0x7f43d4070550 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43d950e640 10 monclient(hunting): picked mon.noname-a con 0x7f43d40713f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43d950e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d4132660 con 0x7f43d40713f0 2024-03-27T03:12:05.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43d950e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d41d8da0 con 0x7f43d4070170 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43d950e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d41d8f30 con 0x7f43d4070550 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43d950e640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43d950e640 10 monclient(hunting): authenticate will time out at 1685.700889s 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43d37fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f43d40713f0 0x7f43d41e09f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:35578/0 (socket says 172.21.15.153:35578) 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43d37fe640 1 -- 172.21.15.153:0/2483179742 learned_addr learned my addr 172.21.15.153:0/2483179742 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43bb7fe640 10 client.?.objecter ms_handle_connect 0x7f43d40713f0 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43bb7fe640 10 client.?.objecter ms_handle_connect 0x7f43d4070170 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.395+0000 7f43bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 10 client.?.objecter ms_handle_connect 0x7f43d4070550 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444272845 0 0) 0x7f43d41d8da0 con 0x7f43d4070170 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:05.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 10 monclient(hunting): my global_id is 4474 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c0004eb0 con 0x7f43d4070170 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006084455 0 0) 0x7f43d4132660 con 0x7f43d40713f0 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 10 monclient(hunting): my global_id is 4440 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.396+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d41d8da0 con 0x7f43d40713f0 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1487115191 0 0) 0x7f43c0004eb0 con 0x7f43d4070170 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43d4132660 con 0x7f43d4070170 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f43c8001bc0 con 0x7f43d4070170 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 10 monclient(hunting): dump: 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:05.397 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3803170731 0 0) 0x7f43d41d8da0 con 0x7f43d40713f0 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43c0004eb0 con 0x7f43d40713f0 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f43cc001cb0 con 0x7f43d40713f0 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.397+0000 7f43bb7fe640 10 monclient(hunting): dump: 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:05.398 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 778188104 0 0) 0x7f43d4132660 con 0x7f43d4070170 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 >> v1:172.21.15.153:6790/0 conn(0x7f43d4070550 legacy=0x7f43d4130af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 >> v1:172.21.15.153:6789/0 conn(0x7f43d40713f0 legacy=0x7f43d41e09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 10 monclient: _finish_hunting 0 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 1 monclient: found mon.b 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:05.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d41365a0 con 0x7f43d4070170 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 10 monclient: _finish_auth 0 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 10 monclient: _check_auth_tickets 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43d950e640 5 monclient: authenticate success, global_id 4474 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43d950e640 10 monclient: _renew_subs 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43d950e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43d950e640 1 -- 172.21.15.153:0/2483179742 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d4136730 con 0x7f43d4070170 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43d950e640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43d950e640 10 monclient: _renew_subs 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43d950e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43d950e640 1 -- 172.21.15.153:0/2483179742 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d4139b20 con 0x7f43d4070170 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43d950e640 1 librados: init done 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43c80038a0 con 0x7f43d4070170 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f43c8003bb0 con 0x7f43d4070170 2024-03-27T03:12:05.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43bb7fe640 10 monclient: dump: 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:05.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.398+0000 7f43d17fa640 4 set_mon_vals no callback set 2024-03-27T03:12:05.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.402+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f43c8019b80 con 0x7f43d4070170 2024-03-27T03:12:05.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.402+0000 7f43d950e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:05.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.402+0000 7f43d950e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:05.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.402+0000 7f43d950e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:05.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.402+0000 7f43d950e640 1 -- 172.21.15.153:0/2483179742 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b4005080 con 0x7f43d4070170 2024-03-27T03:12:05.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.402+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f43c8083940 con 0x7f43d4070170 2024-03-27T03:12:05.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.402+0000 7f43bb7fe640 10 client.4474.objecter ms_dispatch 0x7f43d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:12:05.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.402+0000 7f43bb7fe640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:12:05.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.402+0000 7f43bb7fe640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:12:05.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.402+0000 7f43bb7fe640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-27T03:12:05.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.403+0000 7f43d27fc640 10 monclient: get_auth_request con 0x7f43c0072190 auth_method 0 2024-03-27T03:12:05.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.403+0000 7f43bb7fe640 10 client.4474.objecter ms_handle_connect 0x7f43c0072190 2024-03-27T03:12:05.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.411+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43c804f220 con 0x7f43d4070170 2024-03-27T03:12:05.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.411+0000 7f43bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:05.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.411+0000 7f43bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:12:05.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.816+0000 7f43d950e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:05.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.816+0000 7f43d950e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:05.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.816+0000 7f43d950e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:05.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.816+0000 7f43d950e640 1 -- 172.21.15.153:0/2483179742 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43b4005330 con 0x7f43d4070170 2024-03-27T03:12:05.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.817+0000 7f43bb7fe640 1 -- 172.21.15.153:0/2483179742 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f43c8051150 con 0x7f43d4070170 2024-03-27T03:12:05.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.817+0000 7f43bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:05.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.817+0000 7f43bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:12:05.811 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:12:05.811 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 10 librados: watch_flush enter 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 10 librados: watch_flush exit 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 10 client.4474.objecter successfully canceled tick 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 1 -- 172.21.15.153:0/2483179742 >> v1:172.21.15.153:6816/2637937781 conn(0x7f43c0072190 legacy=0x7f43c0074650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 10 monclient: shutdown 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 1 -- 172.21.15.153:0/2483179742 >> v1:172.21.15.196:6789/0 conn(0x7f43d4070170 legacy=0x7f43d41d8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 1 -- 172.21.15.153:0/2483179742 shutdown_connections 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 1 -- 172.21.15.153:0/2483179742 >> 172.21.15.153:0/2483179742 conn(0x7f43d4067610 msgr2=0x7f43d413d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 1 -- 172.21.15.153:0/2483179742 shutdown_connections 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 1 -- 172.21.15.153:0/2483179742 wait complete. 2024-03-27T03:12:05.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:05.821+0000 7f43d950e640 1 librados: shutdown 2024-03-27T03:12:05.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:12:12.637 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:12:12.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.790+0000 7f4fe359e640 10 monclient: get_monmap_and_config 2024-03-27T03:12:12.783 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.790+0000 7f4fe359e640 10 monclient: build_initial_monmap 2024-03-27T03:12:12.784 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.790+0000 7f4fe359e640 10 monclient: monmap: 2024-03-27T03:12:12.784 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:12.784 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:12.784 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:12:12.791307+0000 2024-03-27T03:12:12.784 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:12:12.791307+0000 2024-03-27T03:12:12.784 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:12.784 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:12.784 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:12.784 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:12:12.784 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:12:12.784 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:12.784 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.791+0000 7f4fe359e640 1 Processor -- start 2024-03-27T03:12:12.785 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.791+0000 7f4fe359e640 1 -- start start 2024-03-27T03:12:12.785 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.791+0000 7f4fe359e640 10 monclient: init 2024-03-27T03:12:12.785 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.792+0000 7f4fe359e640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:12.785 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.792+0000 7f4fe359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:12:12.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.795+0000 7f4fe359e640 10 monclient(hunting): picked mon.noname-b con 0x7f4fe406fd30 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:12.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.795+0000 7f4fe359e640 10 monclient(hunting): picked mon.noname-a con 0x7f4fe4072620 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:12.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.795+0000 7f4fe359e640 10 monclient(hunting): picked mon.noname-c con 0x7f4fe4070110 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.795+0000 7f4fe359e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe4140aa0 con 0x7f4fe4072620 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.795+0000 7f4fe359e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe4071f20 con 0x7f4fe4070110 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.795+0000 7f4fe359e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe4072200 con 0x7f4fe406fd30 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.795+0000 7f4fe359e640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.795+0000 7f4fe359e640 10 monclient(hunting): authenticate will time out at 1693.101474s 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.796+0000 7f4fe259c640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f4fe406fd30 0x7f4fe4140390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:45517/0 (socket says 172.21.15.153:45517) 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.796+0000 7f4fe259c640 1 -- 172.21.15.153:0/3026232648 learned_addr learned my addr 172.21.15.153:0/3026232648 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.796+0000 7f4fe159a640 1 -- 172.21.15.153:0/3026232648 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589188862 0 0) 0x7f4fe4072200 con 0x7f4fe406fd30 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.796+0000 7f4fe159a640 10 monclient(hunting): _negotiate 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.796+0000 7f4fe159a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.796+0000 7f4fe159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.796+0000 7f4fe159a640 10 monclient(hunting): my global_id is 4496 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.796+0000 7f4fe159a640 1 -- 172.21.15.153:0/3026232648 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fcc003520 con 0x7f4fe406fd30 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.796+0000 7f4fe159a640 1 -- 172.21.15.153:0/3026232648 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2477026047 0 0) 0x7f4fcc003520 con 0x7f4fe406fd30 2024-03-27T03:12:12.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.796+0000 7f4fe159a640 1 -- 172.21.15.153:0/3026232648 >> v1:172.21.15.196:6789/0 conn(0x7f4fe4070110 legacy=0x7f4fe414b560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:12:12.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 1 -- 172.21.15.153:0/3026232648 >> v1:172.21.15.153:6789/0 conn(0x7f4fe4072620 legacy=0x7f4fe413cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:12.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 10 monclient: _finish_hunting 0 2024-03-27T03:12:12.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 1 monclient: found mon.noname-b 2024-03-27T03:12:12.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:12.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:12:12.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 1 -- 172.21.15.153:0/3026232648 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe4070ff0 con 0x7f4fe406fd30 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 10 monclient: _finish_auth 0 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 10 monclient: _check_auth_tickets 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 1 -- 172.21.15.153:0/3026232648 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4fd4001690 con 0x7f4fe406fd30 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 10 monclient: dump: 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:12.793 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:12.794 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:12.794 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:12.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe359e640 5 monclient: authenticate success, global_id 4496 2024-03-27T03:12:12.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:12:12.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 1 -- 172.21.15.153:0/3026232648 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fd40017f0 con 0x7f4fe406fd30 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 1 -- 172.21.15.153:0/3026232648 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4fd4001c00 con 0x7f4fe406fd30 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe159a640 10 monclient: dump: 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:12.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe359e640 10 monclient: get_monmap_and_config success 2024-03-27T03:12:12.796 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.799+0000 7f4fe359e640 4 set_mon_vals no callback set 2024-03-27T03:12:12.796 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.800+0000 7f4fe359e640 10 monclient: shutdown 2024-03-27T03:12:12.796 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.800+0000 7f4fe359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:12.796 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.800+0000 7f4fe359e640 1 -- 172.21.15.153:0/3026232648 >> v1:172.21.15.153:6790/0 conn(0x7f4fe406fd30 legacy=0x7f4fe4140390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:12.796 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.801+0000 7f4fe359e640 1 -- 172.21.15.153:0/3026232648 shutdown_connections 2024-03-27T03:12:12.796 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.801+0000 7f4fe359e640 1 -- 172.21.15.153:0/3026232648 >> 172.21.15.153:0/3026232648 conn(0x7f4fe40b59b0 msgr2=0x7f4fe40b5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:12.796 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.802+0000 7f4fe359e640 1 -- 172.21.15.153:0/3026232648 shutdown_connections 2024-03-27T03:12:12.797 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.802+0000 7f4fe359e640 1 -- 172.21.15.153:0/3026232648 wait complete. 2024-03-27T03:12:12.797 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.803+0000 7f4fe359e640 10 monclient: build_initial_monmap 2024-03-27T03:12:12.797 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.803+0000 7f4fe359e640 10 monclient: monmap: 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.804+0000 7f4fe359e640 1 librados: starting msgr at 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.804+0000 7f4fe359e640 1 librados: starting objecter 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.804+0000 7f4fe359e640 1 Processor -- start 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.804+0000 7f4fe359e640 1 -- start start 2024-03-27T03:12:12.798 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.804+0000 7f4fe359e640 1 librados: setting wanted keys 2024-03-27T03:12:12.799 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.804+0000 7f4fe359e640 1 librados: calling monclient init 2024-03-27T03:12:12.799 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.804+0000 7f4fe359e640 10 monclient: init 2024-03-27T03:12:12.799 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.805+0000 7f4fe359e640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:12.799 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.805+0000 7f4fe359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:12:12.799 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.805+0000 7f4fe359e640 10 monclient(hunting): picked mon.noname-c con 0x7f4fe406fd30 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:12.802 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.807+0000 7f4fe359e640 10 monclient(hunting): picked mon.noname-b con 0x7f4fe4070110 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:12.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.807+0000 7f4fe359e640 10 monclient(hunting): picked mon.noname-a con 0x7f4fe4072620 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:12.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.807+0000 7f4fe359e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe40ad660 con 0x7f4fe4072620 2024-03-27T03:12:12.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.807+0000 7f4fe359e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe40ae420 con 0x7f4fe4070110 2024-03-27T03:12:12.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.807+0000 7f4fe359e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe40ad140 con 0x7f4fe406fd30 2024-03-27T03:12:12.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.807+0000 7f4fe359e640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:12.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.807+0000 7f4fe359e640 10 monclient(hunting): authenticate will time out at 1693.113252s 2024-03-27T03:12:12.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.810+0000 7f4fe259c640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f4fe406fd30 0x7f4fe40ac310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:45607/0 (socket says 172.21.15.153:45607) 2024-03-27T03:12:12.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.810+0000 7f4fe259c640 1 -- 172.21.15.153:0/1588990424 learned_addr learned my addr 172.21.15.153:0/1588990424 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:12.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.812+0000 7f4fd2ffd640 10 client.?.objecter ms_handle_connect 0x7f4fe406fd30 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.812+0000 7f4fd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.812+0000 7f4fd2ffd640 10 client.?.objecter ms_handle_connect 0x7f4fe4070110 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.812+0000 7f4fd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078415528 0 0) 0x7f4fe40ad140 con 0x7f4fe406fd30 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 monclient(hunting): my global_id is 4499 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc4004eb0 con 0x7f4fe406fd30 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 client.?.objecter ms_handle_connect 0x7f4fe4072620 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567009882 0 0) 0x7f4fe40ae420 con 0x7f4fe4070110 2024-03-27T03:12:12.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 monclient(hunting): my global_id is 4477 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe40ad140 con 0x7f4fe4070110 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180887853 0 0) 0x7f4fe40ad660 con 0x7f4fe4072620 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 10 monclient(hunting): my global_id is 4446 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe40ae420 con 0x7f4fe4072620 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.813+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2505816083 0 0) 0x7f4fe40ad140 con 0x7f4fe4070110 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.814+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fe40ad660 con 0x7f4fe4070110 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.814+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3018190722 0 0) 0x7f4fe40ae420 con 0x7f4fe4072620 2024-03-27T03:12:12.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.814+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fe40ad140 con 0x7f4fe4072620 2024-03-27T03:12:12.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.814+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4fd8001640 con 0x7f4fe4070110 2024-03-27T03:12:12.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.814+0000 7f4fd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:12.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.815+0000 7f4fd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:12:12.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.815+0000 7f4fd2ffd640 10 monclient(hunting): dump: 2024-03-27T03:12:12.809 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:12.810 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:12.810 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:12.810 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:12.810 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:12.810 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:12.810 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:12.810 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:12.810 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:12.810 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:12.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4fdc001a30 con 0x7f4fe4072620 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 10 monclient(hunting): dump: 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 87939450 0 0) 0x7f4fc4004eb0 con 0x7f4fe406fd30 2024-03-27T03:12:12.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fe40ae420 con 0x7f4fe406fd30 2024-03-27T03:12:12.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3882044703 0 0) 0x7f4fe40ad660 con 0x7f4fe4070110 2024-03-27T03:12:12.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 >> v1:172.21.15.153:6790/0 conn(0x7f4fe406fd30 legacy=0x7f4fe40ac310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:12.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 >> v1:172.21.15.153:6789/0 conn(0x7f4fe4072620 legacy=0x7f4fe41e18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:12.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:12:12.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 1 monclient: found mon.b 2024-03-27T03:12:12.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:12.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:12.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe40acb40 con 0x7f4fe4070110 2024-03-27T03:12:12.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 10 monclient: _finish_auth 0 2024-03-27T03:12:12.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:12:12.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.818+0000 7f4fd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:12.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.819+0000 7f4fe359e640 5 monclient: authenticate success, global_id 4477 2024-03-27T03:12:12.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.819+0000 7f4fe359e640 10 monclient: _renew_subs 2024-03-27T03:12:12.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.819+0000 7f4fe359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:12.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.819+0000 7f4fe359e640 1 -- 172.21.15.153:0/1588990424 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe40acdf0 con 0x7f4fe4070110 2024-03-27T03:12:12.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.819+0000 7f4fe359e640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:12:12.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.819+0000 7f4fe359e640 10 monclient: _renew_subs 2024-03-27T03:12:12.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.819+0000 7f4fe359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:12.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.819+0000 7f4fe359e640 1 -- 172.21.15.153:0/1588990424 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe40af410 con 0x7f4fe4070110 2024-03-27T03:12:12.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.819+0000 7f4fe359e640 1 librados: init done 2024-03-27T03:12:12.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.821+0000 7f4fe359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:12.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.821+0000 7f4fe359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:12.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.821+0000 7f4fe359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:12.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.821+0000 7f4fe359e640 1 -- 172.21.15.153:0/1588990424 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe404f730 con 0x7f4fe4070110 2024-03-27T03:12:12.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.822+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fd8001b50 con 0x7f4fe4070110 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.822+0000 7f4fd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.822+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4fd8001d80 con 0x7f4fe4070110 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.822+0000 7f4fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.822+0000 7f4fd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.822+0000 7f4fd2ffd640 10 monclient: dump: 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:12.816 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:12.817 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.822+0000 7f4fe0d99640 4 set_mon_vals no callback set 2024-03-27T03:12:12.817 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.823+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f4fd8019620 con 0x7f4fe4070110 2024-03-27T03:12:12.817 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.824+0000 7f4fe259c640 10 monclient: get_auth_request con 0x7f4fc406e1c0 auth_method 0 2024-03-27T03:12:12.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.825+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f4fd80834d0 con 0x7f4fe4070110 2024-03-27T03:12:12.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.825+0000 7f4fd2ffd640 10 client.4477.objecter ms_dispatch 0x7f4fe4000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:12:12.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.825+0000 7f4fd2ffd640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:12:12.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.825+0000 7f4fd2ffd640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:12:12.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.825+0000 7f4fd2ffd640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-27T03:12:12.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.825+0000 7f4fd2ffd640 10 client.4477.objecter ms_handle_connect 0x7f4fc406e1c0 2024-03-27T03:12:12.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.829+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4fd804edb0 con 0x7f4fe4070110 2024-03-27T03:12:12.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.829+0000 7f4fd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:12.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:12.829+0000 7f4fd2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:12:13.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.162+0000 7f4fe359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:13.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.162+0000 7f4fe359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:13.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.162+0000 7f4fe359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:13.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.162+0000 7f4fe359e640 1 -- 172.21.15.153:0/1588990424 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fe40b4ef0 con 0x7f4fe4070110 2024-03-27T03:12:13.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.163+0000 7f4fd2ffd640 1 -- 172.21.15.153:0/1588990424 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f4fd8050ce0 con 0x7f4fe4070110 2024-03-27T03:12:13.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.164+0000 7f4fd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:13.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.164+0000 7f4fd2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:12:13.158 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:12:13.159 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:12:13.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.170+0000 7f4fe359e640 10 librados: watch_flush enter 2024-03-27T03:12:13.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.170+0000 7f4fe359e640 10 librados: watch_flush exit 2024-03-27T03:12:13.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.170+0000 7f4fe359e640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-27T03:12:13.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.170+0000 7f4fe359e640 10 client.4477.objecter successfully canceled tick 2024-03-27T03:12:13.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.175+0000 7f4fe359e640 1 -- 172.21.15.153:0/1588990424 >> v1:172.21.15.153:6816/2637937781 conn(0x7f4fc406e1c0 legacy=0x7f4fc4070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:13.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.175+0000 7f4fe359e640 10 monclient: shutdown 2024-03-27T03:12:13.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.175+0000 7f4fe359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:13.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.175+0000 7f4fe359e640 1 -- 172.21.15.153:0/1588990424 >> v1:172.21.15.196:6789/0 conn(0x7f4fe4070110 legacy=0x7f4fe40aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:13.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.175+0000 7f4fe359e640 1 -- 172.21.15.153:0/1588990424 shutdown_connections 2024-03-27T03:12:13.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.175+0000 7f4fe359e640 1 -- 172.21.15.153:0/1588990424 >> 172.21.15.153:0/1588990424 conn(0x7f4fe40b59b0 msgr2=0x7f4fe40b32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:13.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.175+0000 7f4fe359e640 1 -- 172.21.15.153:0/1588990424 shutdown_connections 2024-03-27T03:12:13.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.175+0000 7f4fe359e640 1 -- 172.21.15.153:0/1588990424 wait complete. 2024-03-27T03:12:13.168 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:13.175+0000 7f4fe359e640 1 librados: shutdown 2024-03-27T03:12:13.187 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:12:19.995 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:12:20.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.141+0000 7f452a358640 10 monclient: get_monmap_and_config 2024-03-27T03:12:20.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.141+0000 7f452a358640 10 monclient: build_initial_monmap 2024-03-27T03:12:20.134 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.141+0000 7f452a358640 10 monclient: monmap: 2024-03-27T03:12:20.135 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:20.135 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:20.135 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:12:20.142143+0000 2024-03-27T03:12:20.135 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:12:20.142143+0000 2024-03-27T03:12:20.135 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:20.135 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:20.135 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:20.135 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:12:20.135 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:12:20.135 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:20.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.142+0000 7f452a358640 1 Processor -- start 2024-03-27T03:12:20.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.142+0000 7f452a358640 1 -- start start 2024-03-27T03:12:20.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.142+0000 7f452a358640 10 monclient: init 2024-03-27T03:12:20.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f452a358640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f452a358640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f452a358640 10 monclient(hunting): picked mon.noname-b con 0x7f45240a9750 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f452a358640 10 monclient(hunting): picked mon.noname-c con 0x7f452413e1e0 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f452a358640 10 monclient(hunting): picked mon.noname-a con 0x7f4524071c30 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f452a358640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45240ab300 con 0x7f4524071c30 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f452a358640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45240ac2f0 con 0x7f452413e1e0 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f452a358640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45240ad8d0 con 0x7f45240a9750 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f452a358640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f452a358640 10 monclient(hunting): authenticate will time out at 1700.448840s 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f4523fff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f45240a9750 0x7f452413dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:55792/0 (socket says 172.21.15.153:55792) 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f4523fff640 1 -- 172.21.15.153:0/323527632 learned_addr learned my addr 172.21.15.153:0/323527632 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f4522ffd640 1 -- 172.21.15.153:0/323527632 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304590039 0 0) 0x7f45240ad8d0 con 0x7f45240a9750 2024-03-27T03:12:20.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f4522ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:12:20.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f4522ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:20.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f4522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:20.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f4522ffd640 10 monclient(hunting): my global_id is 4502 2024-03-27T03:12:20.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f4522ffd640 1 -- 172.21.15.153:0/323527632 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4510003440 con 0x7f45240a9750 2024-03-27T03:12:20.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f4522ffd640 1 -- 172.21.15.153:0/323527632 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2324662019 0 0) 0x7f4510003440 con 0x7f45240a9750 2024-03-27T03:12:20.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.143+0000 7f4522ffd640 1 -- 172.21.15.153:0/323527632 >> v1:172.21.15.196:6789/0 conn(0x7f452413e1e0 legacy=0x7f45240ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:12:20.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.144+0000 7f4522ffd640 1 -- 172.21.15.153:0/323527632 >> v1:172.21.15.153:6789/0 conn(0x7f4524071c30 legacy=0x7f45240ace20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.146+0000 7f4522ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.146+0000 7f4522ffd640 1 monclient: found mon.noname-b 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.146+0000 7f4522ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.146+0000 7f4522ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.146+0000 7f4522ffd640 1 -- 172.21.15.153:0/323527632 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45240ada60 con 0x7f45240a9750 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.146+0000 7f4522ffd640 10 monclient: _finish_auth 0 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.146+0000 7f4522ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.146+0000 7f4522ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.146+0000 7f4522ffd640 1 -- 172.21.15.153:0/323527632 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4514001690 con 0x7f45240a9750 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.146+0000 7f4522ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.146+0000 7f4522ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.146+0000 7f4522ffd640 10 monclient: dump: 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:20.143 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.147+0000 7f452a358640 5 monclient: authenticate success, global_id 4502 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.147+0000 7f452a358640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.147+0000 7f4522ffd640 1 -- 172.21.15.153:0/323527632 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4514001860 con 0x7f45240a9750 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.147+0000 7f4522ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.147+0000 7f4522ffd640 1 -- 172.21.15.153:0/323527632 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4514001cb0 con 0x7f45240a9750 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.147+0000 7f4522ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:20.144 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.147+0000 7f4522ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.147+0000 7f4522ffd640 10 monclient: dump: 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.147+0000 7f452a358640 10 monclient: get_monmap_and_config success 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.147+0000 7f452a358640 4 set_mon_vals no callback set 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.148+0000 7f452a358640 10 monclient: shutdown 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.148+0000 7f452a358640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:20.145 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.148+0000 7f452a358640 1 -- 172.21.15.153:0/323527632 >> v1:172.21.15.153:6790/0 conn(0x7f45240a9750 legacy=0x7f452413dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.148+0000 7f452a358640 1 -- 172.21.15.153:0/323527632 shutdown_connections 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.148+0000 7f452a358640 1 -- 172.21.15.153:0/323527632 >> 172.21.15.153:0/323527632 conn(0x7f452406b660 msgr2=0x7f452406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.148+0000 7f452a358640 1 -- 172.21.15.153:0/323527632 shutdown_connections 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.148+0000 7f452a358640 1 -- 172.21.15.153:0/323527632 wait complete. 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.149+0000 7f452a358640 10 monclient: build_initial_monmap 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.149+0000 7f452a358640 10 monclient: monmap: 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:20.146 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:20.147 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:12:20.147 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:12:20.147 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:20.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.149+0000 7f452a358640 1 librados: starting msgr at 2024-03-27T03:12:20.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.149+0000 7f452a358640 1 librados: starting objecter 2024-03-27T03:12:20.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.150+0000 7f452a358640 1 Processor -- start 2024-03-27T03:12:20.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.150+0000 7f452a358640 1 -- start start 2024-03-27T03:12:20.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.150+0000 7f452a358640 1 librados: setting wanted keys 2024-03-27T03:12:20.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.150+0000 7f452a358640 1 librados: calling monclient init 2024-03-27T03:12:20.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.150+0000 7f452a358640 10 monclient: init 2024-03-27T03:12:20.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.151+0000 7f452a358640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:20.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.151+0000 7f452a358640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:12:20.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.151+0000 7f452a358640 10 monclient(hunting): picked mon.noname-c con 0x7f4524071c30 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:20.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.151+0000 7f452a358640 10 monclient(hunting): picked mon.noname-b con 0x7f45240a9750 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:20.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.151+0000 7f452a358640 10 monclient(hunting): picked mon.noname-a con 0x7f452413e1e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:20.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.151+0000 7f452a358640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45240b2600 con 0x7f452413e1e0 2024-03-27T03:12:20.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.151+0000 7f452a358640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45240b28e0 con 0x7f45240a9750 2024-03-27T03:12:20.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.151+0000 7f452a358640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45240b3260 con 0x7f4524071c30 2024-03-27T03:12:20.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.151+0000 7f452a358640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:20.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.151+0000 7f452a358640 10 monclient(hunting): authenticate will time out at 1700.457391s 2024-03-27T03:12:20.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.152+0000 7f4523fff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f4524071c30 0x7f45241da7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:56654/0 (socket says 172.21.15.153:56654) 2024-03-27T03:12:20.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.152+0000 7f4523fff640 1 -- 172.21.15.153:0/2276983194 learned_addr learned my addr 172.21.15.153:0/2276983194 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.152+0000 7f4520ff9640 10 client.?.objecter ms_handle_connect 0x7f4524071c30 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.152+0000 7f4520ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.152+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077934174 0 0) 0x7f45240b3260 con 0x7f4524071c30 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.152+0000 7f4520ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.152+0000 7f4520ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.152+0000 7f4520ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.152+0000 7f4520ff9640 10 monclient(hunting): my global_id is 4505 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.152+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4504004eb0 con 0x7f4524071c30 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 10 client.?.objecter ms_handle_connect 0x7f45240a9750 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107925536 0 0) 0x7f45240b28e0 con 0x7f45240a9750 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:20.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:20.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 10 monclient(hunting): my global_id is 4480 2024-03-27T03:12:20.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45240b3260 con 0x7f45240a9750 2024-03-27T03:12:20.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2858290494 0 0) 0x7f4504004eb0 con 0x7f4524071c30 2024-03-27T03:12:20.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45240b28e0 con 0x7f4524071c30 2024-03-27T03:12:20.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4514001b70 con 0x7f4524071c30 2024-03-27T03:12:20.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:20.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.153+0000 7f4520ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:12:20.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 10 monclient(hunting): dump: 2024-03-27T03:12:20.154 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:20.154 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:20.154 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:20.154 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3084405207 0 0) 0x7f45240b28e0 con 0x7f4524071c30 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 >> v1:172.21.15.196:6789/0 conn(0x7f45240a9750 legacy=0x7f4524134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 >> v1:172.21.15.153:6789/0 conn(0x7f452413e1e0 legacy=0x7f45241e2b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 1 monclient: found mon.c 2024-03-27T03:12:20.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45241426a0 con 0x7f4524071c30 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 10 monclient: _finish_auth 0 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.154+0000 7f4520ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.155+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4514001cd0 con 0x7f4524071c30 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.155+0000 7f4520ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.155+0000 7f45227fc640 4 set_mon_vals no callback set 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.155+0000 7f452a358640 5 monclient: authenticate success, global_id 4505 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.155+0000 7f452a358640 10 monclient: _renew_subs 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.155+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4514004e10 con 0x7f4524071c30 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.155+0000 7f452a358640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:20.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.155+0000 7f452a358640 1 -- 172.21.15.153:0/2276983194 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452413a010 con 0x7f4524071c30 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.155+0000 7f4520ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.155+0000 7f4520ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.155+0000 7f4520ff9640 10 monclient: dump: 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:20.157 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:20.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.156+0000 7f452a358640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:12:20.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.156+0000 7f452a358640 10 monclient: _renew_subs 2024-03-27T03:12:20.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.156+0000 7f452a358640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:20.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.156+0000 7f452a358640 1 -- 172.21.15.153:0/2276983194 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45241355b0 con 0x7f4524071c30 2024-03-27T03:12:20.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.156+0000 7f452a358640 1 librados: init done 2024-03-27T03:12:20.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.157+0000 7f452a358640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:20.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.157+0000 7f452a358640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:20.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.157+0000 7f452a358640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:20.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.157+0000 7f452a358640 1 -- 172.21.15.153:0/2276983194 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4524075f10 con 0x7f4524071c30 2024-03-27T03:12:20.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.159+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f451401ade0 con 0x7f4524071c30 2024-03-27T03:12:20.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.159+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f4514084b90 con 0x7f4524071c30 2024-03-27T03:12:20.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.159+0000 7f4520ff9640 10 client.4505.objecter ms_dispatch 0x7f4524000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:12:20.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.159+0000 7f4520ff9640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:12:20.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.159+0000 7f4520ff9640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:12:20.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.159+0000 7f4520ff9640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-27T03:12:20.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.159+0000 7f45237fe640 10 monclient: get_auth_request con 0x7f450406dd30 auth_method 0 2024-03-27T03:12:20.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.160+0000 7f4520ff9640 10 client.4505.objecter ms_handle_connect 0x7f450406dd30 2024-03-27T03:12:20.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.166+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4514050470 con 0x7f4524071c30 2024-03-27T03:12:20.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.166+0000 7f4520ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:20.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.166+0000 7f4520ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:12:20.461 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.468+0000 7f452a358640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:20.461 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.468+0000 7f452a358640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:20.461 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.468+0000 7f452a358640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:20.461 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.468+0000 7f452a358640 1 -- 172.21.15.153:0/2276983194 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45240a9eb0 con 0x7f4524071c30 2024-03-27T03:12:20.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.469+0000 7f4520ff9640 1 -- 172.21.15.153:0/2276983194 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f45140523a0 con 0x7f4524071c30 2024-03-27T03:12:20.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.469+0000 7f4520ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:20.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.469+0000 7f4520ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:12:20.462 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:12:20.462 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:12:20.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.472+0000 7f452a358640 10 librados: watch_flush enter 2024-03-27T03:12:20.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.472+0000 7f452a358640 10 librados: watch_flush exit 2024-03-27T03:12:20.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.472+0000 7f452a358640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-27T03:12:20.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.472+0000 7f452a358640 10 client.4505.objecter successfully canceled tick 2024-03-27T03:12:20.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.472+0000 7f452a358640 1 -- 172.21.15.153:0/2276983194 >> v1:172.21.15.153:6816/2637937781 conn(0x7f450406dd30 legacy=0x7f45040701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:20.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.472+0000 7f452a358640 10 monclient: shutdown 2024-03-27T03:12:20.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.472+0000 7f452a358640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:20.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.472+0000 7f452a358640 1 -- 172.21.15.153:0/2276983194 >> v1:172.21.15.153:6790/0 conn(0x7f4524071c30 legacy=0x7f45241da7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:20.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.472+0000 7f452a358640 1 -- 172.21.15.153:0/2276983194 shutdown_connections 2024-03-27T03:12:20.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.472+0000 7f452a358640 1 -- 172.21.15.153:0/2276983194 >> 172.21.15.153:0/2276983194 conn(0x7f452406b660 msgr2=0x7f452406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:20.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.473+0000 7f452a358640 1 -- 172.21.15.153:0/2276983194 shutdown_connections 2024-03-27T03:12:20.467 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.473+0000 7f452a358640 1 -- 172.21.15.153:0/2276983194 wait complete. 2024-03-27T03:12:20.467 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:20.473+0000 7f452a358640 1 librados: shutdown 2024-03-27T03:12:20.480 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:12:27.288 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:12:27.437 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.443+0000 7fcde9ed9640 10 monclient: get_monmap_and_config 2024-03-27T03:12:27.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.443+0000 7fcde9ed9640 10 monclient: build_initial_monmap 2024-03-27T03:12:27.438 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.443+0000 7fcde9ed9640 10 monclient: monmap: 2024-03-27T03:12:27.438 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:27.438 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:27.438 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:12:27.444254+0000 2024-03-27T03:12:27.438 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:12:27.444254+0000 2024-03-27T03:12:27.438 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:27.438 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:27.438 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:27.438 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:12:27.438 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:12:27.438 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:27.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.446+0000 7fcde9ed9640 1 Processor -- start 2024-03-27T03:12:27.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.446+0000 7fcde9ed9640 1 -- start start 2024-03-27T03:12:27.439 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.446+0000 7fcde9ed9640 10 monclient: init 2024-03-27T03:12:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.447+0000 7fcde9ed9640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.447+0000 7fcde9ed9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:12:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.447+0000 7fcde9ed9640 10 monclient(hunting): picked mon.noname-b con 0x7fcde40ae250 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.447+0000 7fcde9ed9640 10 monclient(hunting): picked mon.noname-a con 0x7fcde413f970 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.447+0000 7fcde9ed9640 10 monclient(hunting): picked mon.noname-c con 0x7fcde4142800 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.447+0000 7fcde9ed9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde40afe00 con 0x7fcde413f970 2024-03-27T03:12:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.447+0000 7fcde9ed9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde4142c10 con 0x7fcde4142800 2024-03-27T03:12:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.447+0000 7fcde9ed9640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde40afb20 con 0x7fcde40ae250 2024-03-27T03:12:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.447+0000 7fcde9ed9640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:27.440 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.447+0000 7fcde9ed9640 10 monclient(hunting): authenticate will time out at 1707.753111s 2024-03-27T03:12:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.448+0000 7fcde3fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fcde4142800 0x7fcde414da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:36736/0 (socket says 172.21.15.153:36736) 2024-03-27T03:12:27.441 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.448+0000 7fcde3fff640 1 -- 172.21.15.153:0/2734304095 learned_addr learned my addr 172.21.15.153:0/2734304095 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:27.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.451+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376251163 0 0) 0x7fcde4142c10 con 0x7fcde4142800 2024-03-27T03:12:27.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient(hunting): _negotiate 2024-03-27T03:12:27.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:27.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:27.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient(hunting): my global_id is 4483 2024-03-27T03:12:27.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd0003520 con 0x7fcde4142800 2024-03-27T03:12:27.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119371185 0 0) 0x7fcde40afb20 con 0x7fcde40ae250 2024-03-27T03:12:27.446 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient(hunting): _negotiate 2024-03-27T03:12:27.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:27.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:27.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient(hunting): my global_id is 4508 2024-03-27T03:12:27.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde4142c10 con 0x7fcde40ae250 2024-03-27T03:12:27.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035462223 0 0) 0x7fcde40afe00 con 0x7fcde413f970 2024-03-27T03:12:27.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient(hunting): _negotiate 2024-03-27T03:12:27.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:27.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:27.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient(hunting): my global_id is 4449 2024-03-27T03:12:27.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde40afb20 con 0x7fcde413f970 2024-03-27T03:12:27.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2412065079 0 0) 0x7fcde4142c10 con 0x7fcde40ae250 2024-03-27T03:12:27.447 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 >> v1:172.21.15.196:6789/0 conn(0x7fcde4142800 legacy=0x7fcde414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 >> v1:172.21.15.153:6789/0 conn(0x7fcde413f970 legacy=0x7fcde41420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient: _finish_hunting 0 2024-03-27T03:12:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 1 monclient: found mon.noname-b 2024-03-27T03:12:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:12:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde40aab80 con 0x7fcde40ae250 2024-03-27T03:12:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient: _finish_auth 0 2024-03-27T03:12:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient: _check_auth_tickets 2024-03-27T03:12:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fcdcc001670 con 0x7fcde40ae250 2024-03-27T03:12:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:27.448 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient: dump: 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde9ed9640 5 monclient: authenticate success, global_id 4508 2024-03-27T03:12:27.449 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde9ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:12:27.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcdcc0017d0 con 0x7fcde40ae250 2024-03-27T03:12:27.450 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:27.460 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde9ed9640 10 monclient: get_monmap_and_config success 2024-03-27T03:12:27.460 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.452+0000 7fcde9ed9640 4 set_mon_vals no callback set 2024-03-27T03:12:27.460 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.453+0000 7fcde27fc640 1 -- 172.21.15.153:0/2734304095 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fcdcc001be0 con 0x7fcde40ae250 2024-03-27T03:12:27.460 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.453+0000 7fcde27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:27.460 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.453+0000 7fcde27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:12:27.461 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.453+0000 7fcde27fc640 10 monclient: dump: 2024-03-27T03:12:27.461 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:27.461 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:27.461 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:27.461 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:27.461 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:27.461 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:27.461 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:27.461 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:27.461 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:27.461 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.453+0000 7fcde9ed9640 10 monclient: shutdown 2024-03-27T03:12:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.453+0000 7fcde9ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.453+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2734304095 >> v1:172.21.15.153:6790/0 conn(0x7fcde40ae250 legacy=0x7fcde413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.454+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2734304095 shutdown_connections 2024-03-27T03:12:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.454+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2734304095 >> 172.21.15.153:0/2734304095 conn(0x7fcde40b35e0 msgr2=0x7fcde40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.454+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2734304095 shutdown_connections 2024-03-27T03:12:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.454+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2734304095 wait complete. 2024-03-27T03:12:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.455+0000 7fcde9ed9640 10 monclient: build_initial_monmap 2024-03-27T03:12:27.462 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.455+0000 7fcde9ed9640 10 monclient: monmap: 2024-03-27T03:12:27.463 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:27.463 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:12:27.463 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:12:27.463 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:12:27.463 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:27.463 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:27.463 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:27.463 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:12:27.463 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:12:27.463 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:27.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.455+0000 7fcde9ed9640 1 librados: starting msgr at 2024-03-27T03:12:27.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.455+0000 7fcde9ed9640 1 librados: starting objecter 2024-03-27T03:12:27.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.455+0000 7fcde9ed9640 1 Processor -- start 2024-03-27T03:12:27.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.455+0000 7fcde9ed9640 1 -- start start 2024-03-27T03:12:27.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.455+0000 7fcde9ed9640 1 librados: setting wanted keys 2024-03-27T03:12:27.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.455+0000 7fcde9ed9640 1 librados: calling monclient init 2024-03-27T03:12:27.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.455+0000 7fcde9ed9640 10 monclient: init 2024-03-27T03:12:27.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.456+0000 7fcde9ed9640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:27.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.456+0000 7fcde9ed9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:12:27.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.456+0000 7fcde9ed9640 10 monclient(hunting): picked mon.noname-c con 0x7fcde40ae250 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:27.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.456+0000 7fcde9ed9640 10 monclient(hunting): picked mon.noname-b con 0x7fcde413f970 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:27.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.456+0000 7fcde9ed9640 10 monclient(hunting): picked mon.noname-a con 0x7fcde4142800 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:27.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.456+0000 7fcde9ed9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde4141940 con 0x7fcde4142800 2024-03-27T03:12:27.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.456+0000 7fcde9ed9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde40ab9a0 con 0x7fcde413f970 2024-03-27T03:12:27.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.456+0000 7fcde9ed9640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde41d6790 con 0x7fcde40ae250 2024-03-27T03:12:27.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.456+0000 7fcde9ed9640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:27.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.456+0000 7fcde9ed9640 10 monclient(hunting): authenticate will time out at 1707.762324s 2024-03-27T03:12:27.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.459+0000 7fcde2ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fcde413f970 0x7fcde41316e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:38416/0 (socket says 172.21.15.153:38416) 2024-03-27T03:12:27.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.459+0000 7fcde2ffd640 1 -- 172.21.15.153:0/2876833083 learned_addr learned my addr 172.21.15.153:0/2876833083 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:27.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.461+0000 7fcddbfff640 10 client.?.objecter ms_handle_connect 0x7fcde413f970 2024-03-27T03:12:27.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.461+0000 7fcddbfff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:27.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.461+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971192363 0 0) 0x7fcde40ab9a0 con 0x7fcde413f970 2024-03-27T03:12:27.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.461+0000 7fcddbfff640 10 monclient(hunting): _negotiate 2024-03-27T03:12:27.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.461+0000 7fcddbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:27.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.461+0000 7fcddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:27.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.461+0000 7fcddbfff640 10 monclient(hunting): my global_id is 4486 2024-03-27T03:12:27.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.461+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdc4004eb0 con 0x7fcde413f970 2024-03-27T03:12:27.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.461+0000 7fcddbfff640 10 client.?.objecter ms_handle_connect 0x7fcde40ae250 2024-03-27T03:12:27.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.461+0000 7fcddbfff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:27.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.461+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715105874 0 0) 0x7fcde41d6790 con 0x7fcde40ae250 2024-03-27T03:12:27.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.462+0000 7fcddbfff640 10 monclient(hunting): _negotiate 2024-03-27T03:12:27.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.462+0000 7fcddbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:27.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.462+0000 7fcddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:27.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.462+0000 7fcddbfff640 10 monclient(hunting): my global_id is 4511 2024-03-27T03:12:27.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.462+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde40ab9a0 con 0x7fcde40ae250 2024-03-27T03:12:27.467 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.462+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1337194125 0 0) 0x7fcdc4004eb0 con 0x7fcde413f970 2024-03-27T03:12:27.467 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.462+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcde41d6790 con 0x7fcde413f970 2024-03-27T03:12:27.467 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.462+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fcdd4001a10 con 0x7fcde413f970 2024-03-27T03:12:27.467 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.462+0000 7fcddbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:27.467 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.462+0000 7fcddbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:12:27.467 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.462+0000 7fcddbfff640 10 monclient(hunting): dump: 2024-03-27T03:12:27.467 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:27.467 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:27.467 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:27.467 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:27.467 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:27.467 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:27.468 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:27.468 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:27.468 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:27.468 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:27.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.462+0000 7fcddbfff640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:12:27.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.463+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2209328794 0 0) 0x7fcde40ab9a0 con 0x7fcde40ae250 2024-03-27T03:12:27.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.463+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdc4004eb0 con 0x7fcde40ae250 2024-03-27T03:12:27.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.463+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fcdcc001ba0 con 0x7fcde40ae250 2024-03-27T03:12:27.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.463+0000 7fcddbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:27.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.463+0000 7fcddbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:12:27.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.464+0000 7fcddbfff640 10 monclient(hunting): dump: 2024-03-27T03:12:27.469 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:27.469 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:27.469 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:27.469 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:27.469 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:27.469 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:27.469 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:27.469 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:27.469 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:27.469 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:27.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.464+0000 7fcddbfff640 10 client.?.objecter ms_handle_connect 0x7fcde4142800 2024-03-27T03:12:27.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.464+0000 7fcddbfff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:27.470 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.464+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3473548236 0 0) 0x7fcde41d6790 con 0x7fcde413f970 2024-03-27T03:12:27.470 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.464+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 >> v1:172.21.15.153:6790/0 conn(0x7fcde40ae250 legacy=0x7fcde41d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:27.470 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.464+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 >> v1:172.21.15.153:6789/0 conn(0x7fcde4142800 legacy=0x7fcde41de420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:27.470 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.464+0000 7fcddbfff640 10 monclient: _finish_hunting 0 2024-03-27T03:12:27.470 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.464+0000 7fcddbfff640 1 monclient: found mon.b 2024-03-27T03:12:27.470 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.464+0000 7fcddbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:27.470 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.464+0000 7fcddbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:27.470 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.464+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde40ab0a0 con 0x7fcde413f970 2024-03-27T03:12:27.470 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.464+0000 7fcddbfff640 10 monclient: _finish_auth 0 2024-03-27T03:12:27.470 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.465+0000 7fcddbfff640 10 monclient: _check_auth_tickets 2024-03-27T03:12:27.470 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.465+0000 7fcddbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:27.470 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.465+0000 7fcde9ed9640 5 monclient: authenticate success, global_id 4486 2024-03-27T03:12:27.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.465+0000 7fcde9ed9640 10 monclient: _renew_subs 2024-03-27T03:12:27.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.465+0000 7fcde9ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:27.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.465+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2876833083 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde4141e60 con 0x7fcde413f970 2024-03-27T03:12:27.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.465+0000 7fcde9ed9640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:12:27.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.465+0000 7fcde9ed9640 10 monclient: _renew_subs 2024-03-27T03:12:27.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.465+0000 7fcde9ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:27.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.465+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2876833083 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde4131fd0 con 0x7fcde413f970 2024-03-27T03:12:27.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.465+0000 7fcde9ed9640 1 librados: init done 2024-03-27T03:12:27.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.466+0000 7fcde9ed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:27.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.466+0000 7fcde9ed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:27.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.466+0000 7fcde9ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:27.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.466+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2876833083 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde40ab2f0 con 0x7fcde413f970 2024-03-27T03:12:27.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.467+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcdd4001b70 con 0x7fcde413f970 2024-03-27T03:12:27.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.469+0000 7fcddbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:27.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.469+0000 7fcde1ffb640 4 set_mon_vals no callback set 2024-03-27T03:12:27.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.469+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fcdd4003a60 con 0x7fcde413f970 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.469+0000 7fcddbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.469+0000 7fcddbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.469+0000 7fcddbfff640 10 monclient: dump: 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:27.475 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:27.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.469+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fcdd4003c80 con 0x7fcde413f970 2024-03-27T03:12:27.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.470+0000 7fcde37fe640 10 monclient: get_auth_request con 0x7fcdc406dfb0 auth_method 0 2024-03-27T03:12:27.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.470+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fcdd4084ac0 con 0x7fcde413f970 2024-03-27T03:12:27.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.470+0000 7fcddbfff640 10 client.4486.objecter ms_dispatch 0x7fcde4000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:12:27.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.470+0000 7fcddbfff640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:12:27.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.470+0000 7fcddbfff640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:12:27.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.470+0000 7fcddbfff640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-27T03:12:27.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.470+0000 7fcddbfff640 10 client.4486.objecter ms_handle_connect 0x7fcdc406dfb0 2024-03-27T03:12:27.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.477+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcdd4050420 con 0x7fcde413f970 2024-03-27T03:12:27.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.477+0000 7fcddbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:27.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.477+0000 7fcddbfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:12:27.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.800+0000 7fcde9ed9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:27.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.800+0000 7fcde9ed9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:27.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.800+0000 7fcde9ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:27.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.800+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2876833083 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcde40ae910 con 0x7fcde413f970 2024-03-27T03:12:27.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.802+0000 7fcddbfff640 1 -- 172.21.15.153:0/2876833083 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fcdd4052350 con 0x7fcde413f970 2024-03-27T03:12:27.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.802+0000 7fcddbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:27.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.802+0000 7fcddbfff640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:12:27.797 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:12:27.797 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:12:27.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.808+0000 7fcde9ed9640 10 librados: watch_flush enter 2024-03-27T03:12:27.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.809+0000 7fcde9ed9640 10 librados: watch_flush exit 2024-03-27T03:12:27.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.809+0000 7fcde9ed9640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-27T03:12:27.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.809+0000 7fcde9ed9640 10 client.4486.objecter successfully canceled tick 2024-03-27T03:12:27.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.810+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2876833083 >> v1:172.21.15.153:6816/2637937781 conn(0x7fcdc406dfb0 legacy=0x7fcdc4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:27.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.810+0000 7fcde9ed9640 10 monclient: shutdown 2024-03-27T03:12:27.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.810+0000 7fcde9ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:27.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.810+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2876833083 >> v1:172.21.15.196:6789/0 conn(0x7fcde413f970 legacy=0x7fcde41316e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:27.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.813+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2876833083 shutdown_connections 2024-03-27T03:12:27.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.813+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2876833083 >> 172.21.15.153:0/2876833083 conn(0x7fcde40b35e0 msgr2=0x7fcde413e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:27.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.814+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2876833083 shutdown_connections 2024-03-27T03:12:27.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.814+0000 7fcde9ed9640 1 -- 172.21.15.153:0/2876833083 wait complete. 2024-03-27T03:12:27.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:27.814+0000 7fcde9ed9640 1 librados: shutdown 2024-03-27T03:12:27.827 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:12:34.634 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:12:34.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.799+0000 7fe521a40640 10 monclient: get_monmap_and_config 2024-03-27T03:12:34.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.799+0000 7fe521a40640 10 monclient: build_initial_monmap 2024-03-27T03:12:34.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.799+0000 7fe521a40640 10 monclient: monmap: 2024-03-27T03:12:34.793 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:34.793 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:34.793 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:12:34.800965+0000 2024-03-27T03:12:34.793 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:12:34.800965+0000 2024-03-27T03:12:34.793 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:34.793 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:34.794 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:34.794 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:12:34.794 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:12:34.794 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:34.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 1 Processor -- start 2024-03-27T03:12:34.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 1 -- start start 2024-03-27T03:12:34.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 10 monclient: init 2024-03-27T03:12:34.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:34.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:12:34.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 10 monclient(hunting): picked mon.noname-b con 0x7fe51c1457f0 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:34.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 10 monclient(hunting): picked mon.noname-c con 0x7fe51c13f650 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:34.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 10 monclient(hunting): picked mon.noname-a con 0x7fe51c0a9d70 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:34.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51c0aaef0 con 0x7fe51c0a9d70 2024-03-27T03:12:34.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51c145bd0 con 0x7fe51c13f650 2024-03-27T03:12:34.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51c145eb0 con 0x7fe51c1457f0 2024-03-27T03:12:34.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:34.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.800+0000 7fe521a40640 10 monclient(hunting): authenticate will time out at 1715.107560s 2024-03-27T03:12:34.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.807+0000 7fe51b7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe51c0a9d70 0x7fe51c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:42757/0 (socket says 172.21.15.153:42757) 2024-03-27T03:12:34.801 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.807+0000 7fe51b7fe640 1 -- 172.21.15.153:0/2400408020 learned_addr learned my addr 172.21.15.153:0/2400408020 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:34.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.807+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184932834 0 0) 0x7fe51c145eb0 con 0x7fe51c1457f0 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient(hunting): my global_id is 4514 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4fc003520 con 0x7fe51c1457f0 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141152470 0 0) 0x7fe51c145bd0 con 0x7fe51c13f650 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient(hunting): my global_id is 4489 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe51c145eb0 con 0x7fe51c13f650 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821727594 0 0) 0x7fe51c0aaef0 con 0x7fe51c0a9d70 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:12:34.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient(hunting): my global_id is 4455 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe51c145bd0 con 0x7fe51c0a9d70 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3968712755 0 0) 0x7fe51c145eb0 con 0x7fe51c13f650 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 >> v1:172.21.15.153:6790/0 conn(0x7fe51c1457f0 legacy=0x7fe51c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 >> v1:172.21.15.153:6789/0 conn(0x7fe51c0a9d70 legacy=0x7fe51c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 1 monclient: found mon.noname-c 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c146340 con 0x7fe51c13f650 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient: _finish_auth 0 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:12:34.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe50c0015a0 con 0x7fe51c13f650 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.808+0000 7fe51a7fc640 10 monclient: dump: 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:34.809 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.812+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe50c0017c0 con 0x7fe51c13f650 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.812+0000 7fe51a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.812+0000 7fe51a7fc640 1 -- 172.21.15.153:0/2400408020 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe50c001b50 con 0x7fe51c13f650 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.812+0000 7fe51a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.812+0000 7fe51a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.812+0000 7fe51a7fc640 10 monclient: dump: 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:34.810 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:34.811 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:34.811 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:34.811 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:34.811 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:34.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.812+0000 7fe521a40640 5 monclient: authenticate success, global_id 4489 2024-03-27T03:12:34.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.812+0000 7fe521a40640 10 monclient: get_monmap_and_config success 2024-03-27T03:12:34.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.812+0000 7fe521a40640 4 set_mon_vals no callback set 2024-03-27T03:12:34.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.813+0000 7fe521a40640 10 monclient: shutdown 2024-03-27T03:12:34.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.813+0000 7fe521a40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:34.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.813+0000 7fe521a40640 1 -- 172.21.15.153:0/2400408020 >> v1:172.21.15.196:6789/0 conn(0x7fe51c13f650 legacy=0x7fe51c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:34.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.813+0000 7fe521a40640 1 -- 172.21.15.153:0/2400408020 shutdown_connections 2024-03-27T03:12:34.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.813+0000 7fe521a40640 1 -- 172.21.15.153:0/2400408020 >> 172.21.15.153:0/2400408020 conn(0x7fe51c066a00 msgr2=0x7fe51c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:34.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.813+0000 7fe521a40640 1 -- 172.21.15.153:0/2400408020 shutdown_connections 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.813+0000 7fe521a40640 1 -- 172.21.15.153:0/2400408020 wait complete. 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.814+0000 7fe521a40640 10 monclient: build_initial_monmap 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.814+0000 7fe521a40640 10 monclient: monmap: 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.814+0000 7fe521a40640 1 librados: starting msgr at 2024-03-27T03:12:34.814 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.814+0000 7fe521a40640 1 librados: starting objecter 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.815+0000 7fe521a40640 1 Processor -- start 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.815+0000 7fe521a40640 1 -- start start 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.815+0000 7fe521a40640 1 librados: setting wanted keys 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.815+0000 7fe521a40640 1 librados: calling monclient init 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.815+0000 7fe521a40640 10 monclient: init 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.816+0000 7fe521a40640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.816+0000 7fe521a40640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.816+0000 7fe521a40640 10 monclient(hunting): picked mon.noname-a con 0x7fe51c0a9d70 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.816+0000 7fe521a40640 10 monclient(hunting): picked mon.noname-b con 0x7fe51c13f650 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.816+0000 7fe521a40640 10 monclient(hunting): picked mon.noname-c con 0x7fe51c1457f0 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.816+0000 7fe521a40640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51c13a190 con 0x7fe51c0a9d70 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.816+0000 7fe521a40640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51c145300 con 0x7fe51c13f650 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.816+0000 7fe521a40640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51c139660 con 0x7fe51c1457f0 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.816+0000 7fe521a40640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:34.815 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.816+0000 7fe521a40640 10 monclient(hunting): authenticate will time out at 1715.122690s 2024-03-27T03:12:34.817 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.823+0000 7fe513fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe51c13f650 0x7fe51c139a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:42957/0 (socket says 172.21.15.153:42957) 2024-03-27T03:12:34.817 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.823+0000 7fe513fff640 1 -- 172.21.15.153:0/134012672 learned_addr learned my addr 172.21.15.153:0/134012672 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:34.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.826+0000 7fe518ff9640 10 client.?.objecter ms_handle_connect 0x7fe51c13f650 2024-03-27T03:12:34.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.826+0000 7fe518ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:34.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.826+0000 7fe518ff9640 10 client.?.objecter ms_handle_connect 0x7fe51c1457f0 2024-03-27T03:12:34.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.826+0000 7fe518ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:34.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.826+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946440522 0 0) 0x7fe51c145300 con 0x7fe51c13f650 2024-03-27T03:12:34.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.826+0000 7fe518ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:12:34.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.826+0000 7fe518ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:34.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.826+0000 7fe518ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:34.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 10 monclient(hunting): my global_id is 4492 2024-03-27T03:12:34.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe500004eb0 con 0x7fe51c13f650 2024-03-27T03:12:34.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291670085 0 0) 0x7fe51c139660 con 0x7fe51c1457f0 2024-03-27T03:12:34.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:12:34.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:34.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:34.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 10 monclient(hunting): my global_id is 4517 2024-03-27T03:12:34.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe51c145300 con 0x7fe51c1457f0 2024-03-27T03:12:34.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3510299347 0 0) 0x7fe500004eb0 con 0x7fe51c13f650 2024-03-27T03:12:34.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe51c139660 con 0x7fe51c13f650 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe50c001b90 con 0x7fe51c13f650 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.827+0000 7fe518ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 10 monclient(hunting): dump: 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:34.826 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3203657620 0 0) 0x7fe51c139660 con 0x7fe51c13f650 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 >> v1:172.21.15.153:6790/0 conn(0x7fe51c1457f0 legacy=0x7fe51c1dd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 >> v1:172.21.15.153:6789/0 conn(0x7fe51c0a9d70 legacy=0x7fe51c144a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 1 monclient: found mon.b 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c1329c0 con 0x7fe51c13f650 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 10 monclient: _finish_auth 0 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:34.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe50c0038a0 con 0x7fe51c13f650 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fe50c003cb0 con 0x7fe51c13f650 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe518ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.829+0000 7fe518ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.829+0000 7fe518ff9640 10 monclient: dump: 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:34.828 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:34.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.828+0000 7fe519ffb640 4 set_mon_vals no callback set 2024-03-27T03:12:34.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.830+0000 7fe521a40640 5 monclient: authenticate success, global_id 4492 2024-03-27T03:12:34.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.830+0000 7fe521a40640 10 monclient: _renew_subs 2024-03-27T03:12:34.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.830+0000 7fe521a40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:34.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.830+0000 7fe521a40640 1 -- 172.21.15.153:0/134012672 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51c13dc90 con 0x7fe51c13f650 2024-03-27T03:12:34.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.831+0000 7fe521a40640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:12:34.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.831+0000 7fe521a40640 10 monclient: _renew_subs 2024-03-27T03:12:34.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.831+0000 7fe521a40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:34.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.831+0000 7fe521a40640 1 -- 172.21.15.153:0/134012672 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe51c138130 con 0x7fe51c13f650 2024-03-27T03:12:34.830 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.831+0000 7fe521a40640 1 librados: init done 2024-03-27T03:12:34.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.833+0000 7fe521a40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:34.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.833+0000 7fe521a40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:34.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.833+0000 7fe521a40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:34.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.834+0000 7fe521a40640 1 -- 172.21.15.153:0/134012672 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe51c138480 con 0x7fe51c13f650 2024-03-27T03:12:34.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.837+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fe50c003e10 con 0x7fe51c13f650 2024-03-27T03:12:34.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.838+0000 7fe51affd640 10 monclient: get_auth_request con 0x7fe50006dac0 auth_method 0 2024-03-27T03:12:34.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.838+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fe50c084b20 con 0x7fe51c13f650 2024-03-27T03:12:34.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.838+0000 7fe518ff9640 10 client.4492.objecter ms_dispatch 0x7fe51c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:12:34.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.838+0000 7fe518ff9640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:12:34.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.838+0000 7fe518ff9640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:12:34.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.838+0000 7fe518ff9640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-27T03:12:34.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.838+0000 7fe518ff9640 10 client.4492.objecter ms_handle_connect 0x7fe50006dac0 2024-03-27T03:12:34.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.842+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe50c050400 con 0x7fe51c13f650 2024-03-27T03:12:34.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.842+0000 7fe518ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:34.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:34.842+0000 7fe518ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:12:35.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.176+0000 7fe521a40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:35.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.176+0000 7fe521a40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:35.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.176+0000 7fe521a40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:12:35.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.176+0000 7fe521a40640 1 -- 172.21.15.153:0/134012672 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe51c139460 con 0x7fe51c13f650 2024-03-27T03:12:35.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.179+0000 7fe518ff9640 1 -- 172.21.15.153:0/134012672 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fe50c052330 con 0x7fe51c13f650 2024-03-27T03:12:35.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.179+0000 7fe518ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:35.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.179+0000 7fe518ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:12:35.174 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:12:35.174 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:12:35.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.183+0000 7fe521a40640 10 librados: watch_flush enter 2024-03-27T03:12:35.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.183+0000 7fe521a40640 10 librados: watch_flush exit 2024-03-27T03:12:35.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.183+0000 7fe521a40640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-27T03:12:35.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.183+0000 7fe521a40640 10 client.4492.objecter successfully canceled tick 2024-03-27T03:12:35.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.183+0000 7fe521a40640 1 -- 172.21.15.153:0/134012672 >> v1:172.21.15.153:6816/2637937781 conn(0x7fe50006dac0 legacy=0x7fe50006ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:35.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.183+0000 7fe521a40640 10 monclient: shutdown 2024-03-27T03:12:35.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.183+0000 7fe521a40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:35.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.183+0000 7fe521a40640 1 -- 172.21.15.153:0/134012672 >> v1:172.21.15.196:6789/0 conn(0x7fe51c13f650 legacy=0x7fe51c139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:35.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.184+0000 7fe521a40640 1 -- 172.21.15.153:0/134012672 shutdown_connections 2024-03-27T03:12:35.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.184+0000 7fe521a40640 1 -- 172.21.15.153:0/134012672 >> 172.21.15.153:0/134012672 conn(0x7fe51c066a00 msgr2=0x7fe51c13a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:35.181 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.184+0000 7fe521a40640 1 -- 172.21.15.153:0/134012672 shutdown_connections 2024-03-27T03:12:35.181 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.187+0000 7fe521a40640 1 -- 172.21.15.153:0/134012672 wait complete. 2024-03-27T03:12:35.181 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:35.187+0000 7fe521a40640 1 librados: shutdown 2024-03-27T03:12:35.202 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:12:42.009 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:12:42.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.170+0000 7efe1877b640 10 monclient: get_monmap_and_config 2024-03-27T03:12:42.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.170+0000 7efe1877b640 10 monclient: build_initial_monmap 2024-03-27T03:12:42.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.170+0000 7efe1877b640 10 monclient: monmap: 2024-03-27T03:12:42.165 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:42.165 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:42.166 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:12:42.171303+0000 2024-03-27T03:12:42.166 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:12:42.171303+0000 2024-03-27T03:12:42.166 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:42.166 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:42.166 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:42.166 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:12:42.166 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:12:42.166 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:42.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.173+0000 7efe1877b640 1 Processor -- start 2024-03-27T03:12:42.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.173+0000 7efe1877b640 1 -- start start 2024-03-27T03:12:42.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.173+0000 7efe1877b640 10 monclient: init 2024-03-27T03:12:42.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.174+0000 7efe1877b640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:42.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.174+0000 7efe1877b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:12:42.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.174+0000 7efe1877b640 10 monclient(hunting): picked mon.noname-a con 0x7efe100a9930 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:42.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.174+0000 7efe1877b640 10 monclient(hunting): picked mon.noname-c con 0x7efe10149dc0 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:42.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.174+0000 7efe1877b640 10 monclient(hunting): picked mon.noname-b con 0x7efe1014ff40 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:42.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.174+0000 7efe1877b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe100aae60 con 0x7efe100a9930 2024-03-27T03:12:42.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.174+0000 7efe1877b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe100b4310 con 0x7efe10149dc0 2024-03-27T03:12:42.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.174+0000 7efe1877b640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe100afdd0 con 0x7efe1014ff40 2024-03-27T03:12:42.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.174+0000 7efe1877b640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:42.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.174+0000 7efe1877b640 10 monclient(hunting): authenticate will time out at 1722.480019s 2024-03-27T03:12:42.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.177+0000 7efe15d16640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7efe10149dc0 0x7efe1014a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:53693/0 (socket says 172.21.15.153:53693) 2024-03-27T03:12:42.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.177+0000 7efe15d16640 1 -- 172.21.15.153:0/530338376 learned_addr learned my addr 172.21.15.153:0/530338376 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:42.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.177+0000 7efe15515640 1 -- 172.21.15.153:0/530338376 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675164259 0 0) 0x7efe100b4310 con 0x7efe10149dc0 2024-03-27T03:12:42.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.177+0000 7efe15515640 10 monclient(hunting): _negotiate 2024-03-27T03:12:42.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.177+0000 7efe15515640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:42.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.177+0000 7efe15515640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:42.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.177+0000 7efe15515640 10 monclient(hunting): my global_id is 4495 2024-03-27T03:12:42.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.177+0000 7efe15515640 1 -- 172.21.15.153:0/530338376 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdfc003520 con 0x7efe10149dc0 2024-03-27T03:12:42.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.178+0000 7efe15515640 1 -- 172.21.15.153:0/530338376 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2417030482 0 0) 0x7efdfc003520 con 0x7efe10149dc0 2024-03-27T03:12:42.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.178+0000 7efe15515640 1 -- 172.21.15.153:0/530338376 >> v1:172.21.15.153:6790/0 conn(0x7efe1014ff40 legacy=0x7efe10152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:12:42.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 1 -- 172.21.15.153:0/530338376 >> v1:172.21.15.153:6789/0 conn(0x7efe100a9930 legacy=0x7efe10143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:12:42.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 10 monclient: _finish_hunting 0 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 1 monclient: found mon.noname-c 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 1 -- 172.21.15.153:0/530338376 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1014a950 con 0x7efe10149dc0 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 10 monclient: _finish_auth 0 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 10 monclient: _check_auth_tickets 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe1877b640 5 monclient: authenticate success, global_id 4495 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe1877b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 1 -- 172.21.15.153:0/530338376 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7efe08001a10 con 0x7efe10149dc0 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 10 monclient: dump: 2024-03-27T03:12:42.175 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe1877b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 1 -- 172.21.15.153:0/530338376 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe08001ca0 con 0x7efe10149dc0 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe15515640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe1877b640 10 monclient: get_monmap_and_config success 2024-03-27T03:12:42.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.181+0000 7efe1877b640 4 set_mon_vals no callback set 2024-03-27T03:12:42.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.182+0000 7efe1877b640 10 monclient: shutdown 2024-03-27T03:12:42.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.182+0000 7efe1877b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:42.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.182+0000 7efe1877b640 1 -- 172.21.15.153:0/530338376 >> v1:172.21.15.196:6789/0 conn(0x7efe10149dc0 legacy=0x7efe1014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:42.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.184+0000 7efe1877b640 1 -- 172.21.15.153:0/530338376 shutdown_connections 2024-03-27T03:12:42.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.184+0000 7efe1877b640 1 -- 172.21.15.153:0/530338376 >> 172.21.15.153:0/530338376 conn(0x7efe100b35e0 msgr2=0x7efe100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:42.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.185+0000 7efe1877b640 1 -- 172.21.15.153:0/530338376 shutdown_connections 2024-03-27T03:12:42.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.185+0000 7efe1877b640 1 -- 172.21.15.153:0/530338376 wait complete. 2024-03-27T03:12:42.181 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.186+0000 7efe1877b640 10 monclient: build_initial_monmap 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.186+0000 7efe1877b640 10 monclient: monmap: 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.187+0000 7efe1877b640 1 librados: starting msgr at 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.187+0000 7efe1877b640 1 librados: starting objecter 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.187+0000 7efe1877b640 1 Processor -- start 2024-03-27T03:12:42.182 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.187+0000 7efe1877b640 1 -- start start 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.187+0000 7efe1877b640 1 librados: setting wanted keys 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.187+0000 7efe1877b640 1 librados: calling monclient init 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.187+0000 7efe1877b640 10 monclient: init 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.187+0000 7efe1877b640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.188+0000 7efe1877b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.188+0000 7efe1877b640 10 monclient(hunting): picked mon.noname-b con 0x7efe100a9930 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.188+0000 7efe1877b640 10 monclient(hunting): picked mon.noname-c con 0x7efe10149dc0 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.188+0000 7efe1877b640 10 monclient(hunting): picked mon.noname-a con 0x7efe1014ff40 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.188+0000 7efe1877b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe101e4f40 con 0x7efe1014ff40 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.188+0000 7efe1877b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1011c0d0 con 0x7efe100a9930 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.188+0000 7efe1877b640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe10072010 con 0x7efe10149dc0 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.188+0000 7efe1877b640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:42.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.188+0000 7efe1877b640 10 monclient(hunting): authenticate will time out at 1722.493666s 2024-03-27T03:12:42.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.190+0000 7efe15d16640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7efe10149dc0 0x7efe101e4120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:56563/0 (socket says 172.21.15.153:56563) 2024-03-27T03:12:42.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.190+0000 7efe15d16640 1 -- 172.21.15.153:0/3962629306 learned_addr learned my addr 172.21.15.153:0/3962629306 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:42.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.191+0000 7efe06ffd640 10 client.?.objecter ms_handle_connect 0x7efe10149dc0 2024-03-27T03:12:42.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.191+0000 7efe06ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:42.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.191+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099017627 0 0) 0x7efe10072010 con 0x7efe10149dc0 2024-03-27T03:12:42.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.191+0000 7efe06ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:12:42.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.191+0000 7efe06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:42.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.191+0000 7efe06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:42.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.191+0000 7efe06ffd640 10 monclient(hunting): my global_id is 4520 2024-03-27T03:12:42.185 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.191+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdf4004eb0 con 0x7efe10149dc0 2024-03-27T03:12:42.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.194+0000 7efe06ffd640 10 client.?.objecter ms_handle_connect 0x7efe1014ff40 2024-03-27T03:12:42.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.194+0000 7efe06ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:42.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.194+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054135286 0 0) 0x7efe101e4f40 con 0x7efe1014ff40 2024-03-27T03:12:42.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.194+0000 7efe06ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:12:42.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.194+0000 7efe06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:42.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.194+0000 7efe06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:42.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.194+0000 7efe06ffd640 10 monclient(hunting): my global_id is 4458 2024-03-27T03:12:42.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.194+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe10072010 con 0x7efe1014ff40 2024-03-27T03:12:42.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.194+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1980513151 0 0) 0x7efdf4004eb0 con 0x7efe10149dc0 2024-03-27T03:12:42.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.195+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe101e4f40 con 0x7efe10149dc0 2024-03-27T03:12:42.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.195+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7efe08004e60 con 0x7efe10149dc0 2024-03-27T03:12:42.189 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.195+0000 7efe06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.195+0000 7efe06ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.195+0000 7efe06ffd640 10 monclient(hunting): dump: 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:42.190 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.195+0000 7efe06ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:12:42.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 10 client.?.objecter ms_handle_connect 0x7efe100a9930 2024-03-27T03:12:42.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:42.191 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 939845830 0 0) 0x7efe10072010 con 0x7efe1014ff40 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdf4004eb0 con 0x7efe1014ff40 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7efe0c001710 con 0x7efe1014ff40 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 10 monclient(hunting): dump: 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:42.192 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:42.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516519119 0 0) 0x7efe1011c0d0 con 0x7efe100a9930 2024-03-27T03:12:42.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:12:42.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:42.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:42.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 10 monclient(hunting): my global_id is 4498 2024-03-27T03:12:42.193 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.197+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe10072010 con 0x7efe100a9930 2024-03-27T03:12:42.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3376791623 0 0) 0x7efdf4004eb0 con 0x7efe1014ff40 2024-03-27T03:12:42.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 >> v1:172.21.15.153:6790/0 conn(0x7efe10149dc0 legacy=0x7efe101e4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:42.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 >> v1:172.21.15.196:6789/0 conn(0x7efe100a9930 legacy=0x7efe1006f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:42.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe06ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:12:42.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe06ffd640 1 monclient: found mon.a 2024-03-27T03:12:42.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:42.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe06ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:12:42.195 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1011b1c0 con 0x7efe1014ff40 2024-03-27T03:12:42.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe06ffd640 10 monclient: _finish_auth 0 2024-03-27T03:12:42.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe06ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:12:42.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:42.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe1877b640 5 monclient: authenticate success, global_id 4458 2024-03-27T03:12:42.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe1877b640 10 monclient: _renew_subs 2024-03-27T03:12:42.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe1877b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:12:42.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe1877b640 1 -- 172.21.15.153:0/3962629306 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe10071ce0 con 0x7efe1014ff40 2024-03-27T03:12:42.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe1877b640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:12:42.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe1877b640 10 monclient: _renew_subs 2024-03-27T03:12:42.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe1877b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:12:42.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe1877b640 1 -- 172.21.15.153:0/3962629306 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe10071340 con 0x7efe1014ff40 2024-03-27T03:12:42.196 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.201+0000 7efe1877b640 1 librados: init done 2024-03-27T03:12:42.200 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.202+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe0c001c20 con 0x7efe1014ff40 2024-03-27T03:12:42.200 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.202+0000 7efe06ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:42.200 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.202+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7efe0c003790 con 0x7efe1014ff40 2024-03-27T03:12:42.200 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.202+0000 7efe06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:42.200 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.202+0000 7efe06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:12:42.200 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.202+0000 7efe06ffd640 10 monclient: dump: 2024-03-27T03:12:42.200 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:42.200 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:42.200 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:42.200 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.203+0000 7efe1877b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.203+0000 7efe1877b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.203+0000 7efe1877b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.203+0000 7efe1877b640 1 -- 172.21.15.153:0/3962629306 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe100721a0 con 0x7efe1014ff40 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.204+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7efe0c019760 con 0x7efe1014ff40 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.205+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7efe0c083680 con 0x7efe1014ff40 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.205+0000 7efe06ffd640 10 client.4458.objecter ms_dispatch 0x7efe10000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.205+0000 7efe06ffd640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:12:42.201 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.205+0000 7efe06ffd640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:12:42.202 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.205+0000 7efe06ffd640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-27T03:12:42.202 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.205+0000 7efe14d14640 4 set_mon_vals no callback set 2024-03-27T03:12:42.202 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.205+0000 7efe16517640 10 monclient: get_auth_request con 0x7efdf406dd90 auth_method 0 2024-03-27T03:12:42.202 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.205+0000 7efe06ffd640 10 client.4458.objecter ms_handle_connect 0x7efdf406dd90 2024-03-27T03:12:42.206 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.213+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe0c04ef60 con 0x7efe1014ff40 2024-03-27T03:12:42.206 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.213+0000 7efe06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:42.206 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.213+0000 7efe06ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:12:42.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.547+0000 7efe1877b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:42.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.547+0000 7efe1877b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:42.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.547+0000 7efe1877b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:12:42.540 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.547+0000 7efe1877b640 1 -- 172.21.15.153:0/3962629306 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe10075f10 con 0x7efe1014ff40 2024-03-27T03:12:42.543 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.550+0000 7efe06ffd640 1 -- 172.21.15.153:0/3962629306 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7efe0c050e90 con 0x7efe1014ff40 2024-03-27T03:12:42.543 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.550+0000 7efe06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:42.543 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.550+0000 7efe06ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:12:42.543 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:12:42.544 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:12:42.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.553+0000 7efe1877b640 10 librados: watch_flush enter 2024-03-27T03:12:42.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.553+0000 7efe1877b640 10 librados: watch_flush exit 2024-03-27T03:12:42.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.553+0000 7efe1877b640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-27T03:12:42.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.553+0000 7efe1877b640 10 client.4458.objecter successfully canceled tick 2024-03-27T03:12:42.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.554+0000 7efe1877b640 1 -- 172.21.15.153:0/3962629306 >> v1:172.21.15.153:6816/2637937781 conn(0x7efdf406dd90 legacy=0x7efdf4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:42.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.554+0000 7efe1877b640 10 monclient: shutdown 2024-03-27T03:12:42.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.554+0000 7efe1877b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:42.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.554+0000 7efe1877b640 1 -- 172.21.15.153:0/3962629306 >> v1:172.21.15.153:6789/0 conn(0x7efe1014ff40 legacy=0x7efe101e4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:42.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.554+0000 7efe1877b640 1 -- 172.21.15.153:0/3962629306 shutdown_connections 2024-03-27T03:12:42.547 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.554+0000 7efe1877b640 1 -- 172.21.15.153:0/3962629306 >> 172.21.15.153:0/3962629306 conn(0x7efe100b35e0 msgr2=0x7efe100b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:42.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.554+0000 7efe1877b640 1 -- 172.21.15.153:0/3962629306 shutdown_connections 2024-03-27T03:12:42.551 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.554+0000 7efe1877b640 1 -- 172.21.15.153:0/3962629306 wait complete. 2024-03-27T03:12:42.551 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:42.555+0000 7efe1877b640 1 librados: shutdown 2024-03-27T03:12:42.570 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:12:49.376 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:12:49.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.530+0000 7f1029cd0640 10 monclient: get_monmap_and_config 2024-03-27T03:12:49.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.530+0000 7f1029cd0640 10 monclient: build_initial_monmap 2024-03-27T03:12:49.524 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.530+0000 7f1029cd0640 10 monclient: monmap: 2024-03-27T03:12:49.524 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:49.524 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:49.524 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:12:49.531797+0000 2024-03-27T03:12:49.524 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:12:49.531797+0000 2024-03-27T03:12:49.524 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:49.524 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:49.525 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:49.525 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:12:49.525 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:12:49.525 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:49.525 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.532+0000 7f1029cd0640 1 Processor -- start 2024-03-27T03:12:49.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.532+0000 7f1029cd0640 1 -- start start 2024-03-27T03:12:49.526 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.532+0000 7f1029cd0640 10 monclient: init 2024-03-27T03:12:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.533+0000 7f1029cd0640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.533+0000 7f1029cd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:12:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.533+0000 7f1029cd0640 10 monclient(hunting): picked mon.noname-c con 0x7f101c0389e0 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.533+0000 7f1029cd0640 10 monclient(hunting): picked mon.noname-a con 0x7f101c038dc0 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.533+0000 7f1029cd0640 10 monclient(hunting): picked mon.noname-b con 0x7f101c030660 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.533+0000 7f1029cd0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c0c54c0 con 0x7f101c038dc0 2024-03-27T03:12:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.533+0000 7f1029cd0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c036430 con 0x7f101c0389e0 2024-03-27T03:12:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.533+0000 7f1029cd0640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c039840 con 0x7f101c030660 2024-03-27T03:12:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.533+0000 7f1029cd0640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:49.527 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.533+0000 7f1029cd0640 10 monclient(hunting): authenticate will time out at 1729.838961s 2024-03-27T03:12:49.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.536+0000 7f1023fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f101c038dc0 0x7f101c0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:38429/0 (socket says 172.21.15.153:38429) 2024-03-27T03:12:49.529 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.536+0000 7f1023fff640 1 -- 172.21.15.153:0/2438051673 learned_addr learned my addr 172.21.15.153:0/2438051673 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.537+0000 7f10237fe640 1 -- 172.21.15.153:0/2438051673 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229760646 0 0) 0x7f101c0c54c0 con 0x7f101c038dc0 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.537+0000 7f10237fe640 10 monclient(hunting): _negotiate 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.537+0000 7f10237fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.537+0000 7f10237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.537+0000 7f10237fe640 10 monclient(hunting): my global_id is 4461 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.537+0000 7f10237fe640 1 -- 172.21.15.153:0/2438051673 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1014003520 con 0x7f101c038dc0 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.537+0000 7f10237fe640 1 -- 172.21.15.153:0/2438051673 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309553210 0 0) 0x7f101c039840 con 0x7f101c030660 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.537+0000 7f10237fe640 10 monclient(hunting): _negotiate 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.537+0000 7f10237fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.537+0000 7f10237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.537+0000 7f10237fe640 10 monclient(hunting): my global_id is 4523 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.537+0000 7f10237fe640 1 -- 172.21.15.153:0/2438051673 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c0c54c0 con 0x7f101c030660 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 1 -- 172.21.15.153:0/2438051673 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2864691372 0 0) 0x7f101c0c54c0 con 0x7f101c030660 2024-03-27T03:12:49.532 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 1 -- 172.21.15.153:0/2438051673 >> v1:172.21.15.196:6789/0 conn(0x7f101c0389e0 legacy=0x7f101c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 1 -- 172.21.15.153:0/2438051673 >> v1:172.21.15.153:6789/0 conn(0x7f101c038dc0 legacy=0x7f101c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 10 monclient: _finish_hunting 0 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 1 monclient: found mon.noname-b 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 1 -- 172.21.15.153:0/2438051673 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c035690 con 0x7f101c030660 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 10 monclient: _finish_auth 0 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 10 monclient: _check_auth_tickets 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 1 -- 172.21.15.153:0/2438051673 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f102404f730 con 0x7f101c030660 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f10237fe640 10 monclient: dump: 2024-03-27T03:12:49.533 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:49.534 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:49.534 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:49.534 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:49.534 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:49.534 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:49.534 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:49.534 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:49.534 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:49.534 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:49.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f1029cd0640 5 monclient: authenticate success, global_id 4523 2024-03-27T03:12:49.534 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.538+0000 7f1029cd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:12:49.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.541+0000 7f10237fe640 1 -- 172.21.15.153:0/2438051673 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1024075f10 con 0x7f101c030660 2024-03-27T03:12:49.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.541+0000 7f10237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:49.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.541+0000 7f10237fe640 1 -- 172.21.15.153:0/2438051673 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f10240758b0 con 0x7f101c030660 2024-03-27T03:12:49.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.541+0000 7f10237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:49.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.541+0000 7f10237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:12:49.536 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.541+0000 7f10237fe640 10 monclient: dump: 2024-03-27T03:12:49.536 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:49.536 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:49.536 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:49.536 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.541+0000 7f1029cd0640 10 monclient: get_monmap_and_config success 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.542+0000 7f1029cd0640 4 set_mon_vals no callback set 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.542+0000 7f1029cd0640 10 monclient: shutdown 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.542+0000 7f1029cd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.542+0000 7f1029cd0640 1 -- 172.21.15.153:0/2438051673 >> v1:172.21.15.153:6790/0 conn(0x7f101c030660 legacy=0x7f101c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.543+0000 7f1029cd0640 1 -- 172.21.15.153:0/2438051673 shutdown_connections 2024-03-27T03:12:49.537 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.543+0000 7f1029cd0640 1 -- 172.21.15.153:0/2438051673 >> 172.21.15.153:0/2438051673 conn(0x7f101c005c00 msgr2=0x7f101c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:49.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.544+0000 7f1029cd0640 1 -- 172.21.15.153:0/2438051673 shutdown_connections 2024-03-27T03:12:49.538 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.546+0000 7f1029cd0640 1 -- 172.21.15.153:0/2438051673 wait complete. 2024-03-27T03:12:49.541 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.548+0000 7f1029cd0640 10 monclient: build_initial_monmap 2024-03-27T03:12:49.541 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.548+0000 7f1029cd0640 10 monclient: monmap: 2024-03-27T03:12:49.541 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:49.541 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:12:49.541 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:12:49.541 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:12:49.541 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:49.541 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:49.541 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:49.541 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:12:49.542 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:12:49.542 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:49.542 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.548+0000 7f1029cd0640 1 librados: starting msgr at 2024-03-27T03:12:49.542 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.548+0000 7f1029cd0640 1 librados: starting objecter 2024-03-27T03:12:49.542 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.549+0000 7f1029cd0640 1 Processor -- start 2024-03-27T03:12:49.543 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.549+0000 7f1029cd0640 1 -- start start 2024-03-27T03:12:49.543 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.549+0000 7f1029cd0640 1 librados: setting wanted keys 2024-03-27T03:12:49.543 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.549+0000 7f1029cd0640 1 librados: calling monclient init 2024-03-27T03:12:49.543 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.549+0000 7f1029cd0640 10 monclient: init 2024-03-27T03:12:49.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.550+0000 7f1029cd0640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:49.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.550+0000 7f1029cd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:12:49.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.550+0000 7f1029cd0640 10 monclient(hunting): picked mon.noname-b con 0x7f101c030660 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:49.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.550+0000 7f1029cd0640 10 monclient(hunting): picked mon.noname-a con 0x7f101c0389e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:49.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.550+0000 7f1029cd0640 10 monclient(hunting): picked mon.noname-c con 0x7f101c0bba50 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:49.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.552+0000 7f1029cd0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c0c9060 con 0x7f101c0389e0 2024-03-27T03:12:49.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.552+0000 7f1029cd0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c0c0ae0 con 0x7f101c030660 2024-03-27T03:12:49.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.552+0000 7f1029cd0640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c0c0c70 con 0x7f101c0bba50 2024-03-27T03:12:49.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.552+0000 7f1029cd0640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:49.546 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.552+0000 7f1029cd0640 10 monclient(hunting): authenticate will time out at 1729.858575s 2024-03-27T03:12:49.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.553+0000 7f10294cf640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f101c0bba50 0x7f101c15d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:40849/0 (socket says 172.21.15.153:40849) 2024-03-27T03:12:49.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.554+0000 7f10294cf640 1 -- 172.21.15.153:0/2897495684 learned_addr learned my addr 172.21.15.153:0/2897495684 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:49.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.554+0000 7f10217fa640 10 client.?.objecter ms_handle_connect 0x7f101c0bba50 2024-03-27T03:12:49.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.554+0000 7f10217fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:49.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.554+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112518452 0 0) 0x7f101c0c0c70 con 0x7f101c0bba50 2024-03-27T03:12:49.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.554+0000 7f10217fa640 10 monclient(hunting): _negotiate 2024-03-27T03:12:49.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.554+0000 7f10217fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:49.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.554+0000 7f10217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:49.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.554+0000 7f10217fa640 10 monclient(hunting): my global_id is 4526 2024-03-27T03:12:49.548 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.554+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1008004eb0 con 0x7f101c0bba50 2024-03-27T03:12:49.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.557+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2896346897 0 0) 0x7f1008004eb0 con 0x7f101c0bba50 2024-03-27T03:12:49.550 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.557+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f101c0c0c70 con 0x7f101c0bba50 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.557+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f102404f400 con 0x7f101c0bba50 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.557+0000 7f10217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.557+0000 7f10217fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.557+0000 7f10217fa640 10 monclient(hunting): dump: 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:49.551 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:49.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.557+0000 7f10217fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:12:49.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2915674160 0 0) 0x7f101c0c0c70 con 0x7f101c0bba50 2024-03-27T03:12:49.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 >> v1:172.21.15.196:6789/0 conn(0x7f101c030660 legacy=0x7f101c0bb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:49.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 >> v1:172.21.15.153:6789/0 conn(0x7f101c0389e0 legacy=0x7f101c15c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:49.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 10 monclient: _finish_hunting 0 2024-03-27T03:12:49.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 1 monclient: found mon.c 2024-03-27T03:12:49.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:49.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:49.552 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c0c0e00 con 0x7f101c0bba50 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 10 monclient: _finish_auth 0 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 10 monclient: _check_auth_tickets 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f1029cd0640 5 monclient: authenticate success, global_id 4526 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f1029cd0640 10 monclient: _renew_subs 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f1029cd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f1029cd0640 1 -- 172.21.15.153:0/2897495684 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101c0c89b0 con 0x7f101c0bba50 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f1029cd0640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f1029cd0640 10 monclient: _renew_subs 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f1029cd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f1029cd0640 1 -- 172.21.15.153:0/2897495684 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f101c0c4720 con 0x7f101c0bba50 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f1029cd0640 1 librados: init done 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f102404fdd0 con 0x7f101c0bba50 2024-03-27T03:12:49.553 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f1024067bf0 con 0x7f101c0bba50 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.558+0000 7f10217fa640 10 monclient: dump: 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:49.554 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:49.556 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.559+0000 7f10227fc640 4 set_mon_vals no callback set 2024-03-27T03:12:49.556 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.560+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f1024067d50 con 0x7f101c0bba50 2024-03-27T03:12:49.556 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.561+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f10240ff600 con 0x7f101c0bba50 2024-03-27T03:12:49.556 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.561+0000 7f10217fa640 10 client.4526.objecter ms_dispatch 0x7f1024000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:12:49.556 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.561+0000 7f10217fa640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:12:49.556 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.561+0000 7f10217fa640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:12:49.556 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.561+0000 7f10217fa640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-27T03:12:49.556 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.561+0000 7f1029cd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:49.557 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.562+0000 7f1029cd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:49.557 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.562+0000 7f1029cd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:49.557 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.562+0000 7f1029cd0640 1 -- 172.21.15.153:0/2897495684 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1004005080 con 0x7f101c0bba50 2024-03-27T03:12:49.557 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.562+0000 7f1028cce640 10 monclient: get_auth_request con 0x7f100806db10 auth_method 0 2024-03-27T03:12:49.557 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.563+0000 7f10217fa640 10 client.4526.objecter ms_handle_connect 0x7f100806db10 2024-03-27T03:12:49.563 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.570+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10240caee0 con 0x7f101c0bba50 2024-03-27T03:12:49.564 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.571+0000 7f10217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:49.564 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.571+0000 7f10217fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:12:49.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.892+0000 7f1029cd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:49.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.892+0000 7f1029cd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:49.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.892+0000 7f1029cd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:49.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.892+0000 7f1029cd0640 1 -- 172.21.15.153:0/2897495684 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1004005330 con 0x7f101c0bba50 2024-03-27T03:12:49.885 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.893+0000 7f10217fa640 1 -- 172.21.15.153:0/2897495684 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f10240cce10 con 0x7f101c0bba50 2024-03-27T03:12:49.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.893+0000 7f10217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:49.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.893+0000 7f10217fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:12:49.886 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:12:49.886 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:12:49.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.896+0000 7f1029cd0640 10 librados: watch_flush enter 2024-03-27T03:12:49.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.897+0000 7f1029cd0640 10 librados: watch_flush exit 2024-03-27T03:12:49.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.897+0000 7f1029cd0640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-27T03:12:49.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.897+0000 7f1029cd0640 10 client.4526.objecter successfully canceled tick 2024-03-27T03:12:49.890 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.897+0000 7f1029cd0640 1 -- 172.21.15.153:0/2897495684 >> v1:172.21.15.153:6816/2637937781 conn(0x7f100806db10 legacy=0x7f100806ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:49.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.897+0000 7f1029cd0640 10 monclient: shutdown 2024-03-27T03:12:49.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.897+0000 7f1029cd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:49.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.897+0000 7f1029cd0640 1 -- 172.21.15.153:0/2897495684 >> v1:172.21.15.153:6790/0 conn(0x7f101c0bba50 legacy=0x7f101c15d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:49.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.897+0000 7f1029cd0640 1 -- 172.21.15.153:0/2897495684 shutdown_connections 2024-03-27T03:12:49.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.897+0000 7f1029cd0640 1 -- 172.21.15.153:0/2897495684 >> 172.21.15.153:0/2897495684 conn(0x7f101c005c00 msgr2=0x7f101c03b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:49.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.898+0000 7f1029cd0640 1 -- 172.21.15.153:0/2897495684 shutdown_connections 2024-03-27T03:12:49.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.898+0000 7f1029cd0640 1 -- 172.21.15.153:0/2897495684 wait complete. 2024-03-27T03:12:49.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:49.898+0000 7f1029cd0640 1 librados: shutdown 2024-03-27T03:12:49.910 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:12:56.717 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:12:56.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.880+0000 7ff048c8d640 10 monclient: get_monmap_and_config 2024-03-27T03:12:56.874 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.880+0000 7ff048c8d640 10 monclient: build_initial_monmap 2024-03-27T03:12:56.875 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.880+0000 7ff048c8d640 10 monclient: monmap: 2024-03-27T03:12:56.875 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:56.875 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:56.875 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:12:56.881520+0000 2024-03-27T03:12:56.875 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:12:56.881520+0000 2024-03-27T03:12:56.875 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:56.876 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:56.876 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:56.876 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:12:56.876 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:12:56.876 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:56.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.881+0000 7ff048c8d640 1 Processor -- start 2024-03-27T03:12:56.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.882+0000 7ff048c8d640 1 -- start start 2024-03-27T03:12:56.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.882+0000 7ff048c8d640 10 monclient: init 2024-03-27T03:12:56.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.883+0000 7ff048c8d640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:56.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.883+0000 7ff048c8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:12:56.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.883+0000 7ff048c8d640 10 monclient(hunting): picked mon.noname-a con 0x7ff0440b0860 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:56.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.883+0000 7ff048c8d640 10 monclient(hunting): picked mon.noname-c con 0x7ff04413ef30 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:56.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.883+0000 7ff048c8d640 10 monclient(hunting): picked mon.noname-b con 0x7ff0440af8e0 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:56.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.883+0000 7ff048c8d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0440b0c40 con 0x7ff0440b0860 2024-03-27T03:12:56.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.883+0000 7ff048c8d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0440aed10 con 0x7ff04413ef30 2024-03-27T03:12:56.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.883+0000 7ff048c8d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0440afd50 con 0x7ff0440af8e0 2024-03-27T03:12:56.879 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.883+0000 7ff048c8d640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:56.880 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.883+0000 7ff048c8d640 10 monclient(hunting): authenticate will time out at 1737.188772s 2024-03-27T03:12:56.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.886+0000 7ff04259c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff0440b0860 0x7ff04413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60907/0 (socket says 172.21.15.153:60907) 2024-03-27T03:12:56.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.886+0000 7ff04259c640 1 -- 172.21.15.153:0/1367775857 learned_addr learned my addr 172.21.15.153:0/1367775857 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:56.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.888+0000 7ff04159a640 1 -- 172.21.15.153:0/1367775857 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426874527 0 0) 0x7ff0440afd50 con 0x7ff0440af8e0 2024-03-27T03:12:56.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.888+0000 7ff04159a640 10 monclient(hunting): _negotiate 2024-03-27T03:12:56.882 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.888+0000 7ff04159a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:56.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.888+0000 7ff04159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:56.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.888+0000 7ff04159a640 10 monclient(hunting): my global_id is 4529 2024-03-27T03:12:56.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.888+0000 7ff04159a640 1 -- 172.21.15.153:0/1367775857 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff02c003500 con 0x7ff0440af8e0 2024-03-27T03:12:56.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.888+0000 7ff04159a640 1 -- 172.21.15.153:0/1367775857 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1669461251 0 0) 0x7ff02c003500 con 0x7ff0440af8e0 2024-03-27T03:12:56.883 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.888+0000 7ff04159a640 1 -- 172.21.15.153:0/1367775857 >> v1:172.21.15.196:6789/0 conn(0x7ff04413ef30 legacy=0x7ff0440af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:12:56.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 1 -- 172.21.15.153:0/1367775857 >> v1:172.21.15.153:6789/0 conn(0x7ff0440b0860 legacy=0x7ff04413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:56.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 10 monclient: _finish_hunting 0 2024-03-27T03:12:56.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 1 monclient: found mon.noname-b 2024-03-27T03:12:56.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:56.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:12:56.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 1 -- 172.21.15.153:0/1367775857 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0440ab1b0 con 0x7ff0440af8e0 2024-03-27T03:12:56.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 10 monclient: _finish_auth 0 2024-03-27T03:12:56.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 10 monclient: _check_auth_tickets 2024-03-27T03:12:56.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:56.886 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff048c8d640 5 monclient: authenticate success, global_id 4529 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 1 -- 172.21.15.153:0/1367775857 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff038001690 con 0x7ff0440af8e0 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff048c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 10 monclient: dump: 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:56.887 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff048c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 1 -- 172.21.15.153:0/1367775857 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff038001860 con 0x7ff0440af8e0 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 1 -- 172.21.15.153:0/1367775857 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff038001cb0 con 0x7ff0440af8e0 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff04159a640 10 monclient: dump: 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:56.888 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:56.889 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:56.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff048c8d640 10 monclient: get_monmap_and_config success 2024-03-27T03:12:56.889 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.892+0000 7ff048c8d640 4 set_mon_vals no callback set 2024-03-27T03:12:56.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.893+0000 7ff048c8d640 10 monclient: shutdown 2024-03-27T03:12:56.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.893+0000 7ff048c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:56.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.893+0000 7ff048c8d640 1 -- 172.21.15.153:0/1367775857 >> v1:172.21.15.153:6790/0 conn(0x7ff0440af8e0 legacy=0x7ff044149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:56.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.894+0000 7ff048c8d640 1 -- 172.21.15.153:0/1367775857 shutdown_connections 2024-03-27T03:12:56.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.894+0000 7ff048c8d640 1 -- 172.21.15.153:0/1367775857 >> 172.21.15.153:0/1367775857 conn(0x7ff04406b660 msgr2=0x7ff04406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:56.891 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.894+0000 7ff048c8d640 1 -- 172.21.15.153:0/1367775857 shutdown_connections 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.894+0000 7ff048c8d640 1 -- 172.21.15.153:0/1367775857 wait complete. 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.895+0000 7ff048c8d640 10 monclient: build_initial_monmap 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.895+0000 7ff048c8d640 10 monclient: monmap: 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.895+0000 7ff048c8d640 1 librados: starting msgr at 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.895+0000 7ff048c8d640 1 librados: starting objecter 2024-03-27T03:12:56.892 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.895+0000 7ff048c8d640 1 Processor -- start 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.896+0000 7ff048c8d640 1 -- start start 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.896+0000 7ff048c8d640 1 librados: setting wanted keys 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.896+0000 7ff048c8d640 1 librados: calling monclient init 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.896+0000 7ff048c8d640 10 monclient: init 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.897+0000 7ff048c8d640 10 monclient: _reopen_session rank -1 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.897+0000 7ff048c8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.897+0000 7ff048c8d640 10 monclient(hunting): picked mon.noname-a con 0x7ff0440af8e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.897+0000 7ff048c8d640 10 monclient(hunting): picked mon.noname-c con 0x7ff0440b0860 addr v1:172.21.15.153:6790/0 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.897+0000 7ff048c8d640 10 monclient(hunting): picked mon.noname-b con 0x7ff04413ef30 addr v1:172.21.15.196:6789/0 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.897+0000 7ff048c8d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff044137b50 con 0x7ff0440af8e0 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.897+0000 7ff048c8d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff044137e30 con 0x7ff04413ef30 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.897+0000 7ff048c8d640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff044138110 con 0x7ff0440b0860 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.897+0000 7ff048c8d640 10 monclient(hunting): _renew_subs 2024-03-27T03:12:56.893 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.897+0000 7ff048c8d640 10 monclient(hunting): authenticate will time out at 1737.202813s 2024-03-27T03:12:56.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.902+0000 7ff041d9b640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7ff0440b0860 0x7ff0441ded70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:48219/0 (socket says 172.21.15.153:48219) 2024-03-27T03:12:56.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.902+0000 7ff041d9b640 1 -- 172.21.15.153:0/3833242985 learned_addr learned my addr 172.21.15.153:0/3833242985 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:12:56.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.902+0000 7ff032ffd640 10 client.?.objecter ms_handle_connect 0x7ff0440b0860 2024-03-27T03:12:56.895 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.902+0000 7ff032ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:56.899 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.902+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179104103 0 0) 0x7ff044138110 con 0x7ff0440b0860 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.902+0000 7ff032ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.902+0000 7ff032ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.902+0000 7ff032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.902+0000 7ff032ffd640 10 monclient(hunting): my global_id is 4532 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.902+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff024004eb0 con 0x7ff0440b0860 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 10 client.?.objecter ms_handle_connect 0x7ff0440af8e0 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1756859545 0 0) 0x7ff024004eb0 con 0x7ff0440b0860 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff024005300 con 0x7ff0440b0860 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff03c001b10 con 0x7ff0440b0860 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 10 monclient(hunting): dump: 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:56.900 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256203157 0 0) 0x7ff044137b50 con 0x7ff0440af8e0 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 10 monclient(hunting): my global_id is 4467 2024-03-27T03:12:56.901 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.903+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff024004eb0 con 0x7ff0440af8e0 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2550851406 0 0) 0x7ff024005300 con 0x7ff0440b0860 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 >> v1:172.21.15.196:6789/0 conn(0x7ff04413ef30 legacy=0x7ff0441e6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 >> v1:172.21.15.153:6789/0 conn(0x7ff0440af8e0 legacy=0x7ff0441c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff032ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff032ffd640 1 monclient: found mon.c 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff032ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff032ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0441365e0 con 0x7ff0440b0860 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff032ffd640 10 monclient: _finish_auth 0 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff032ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff032ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff048c8d640 5 monclient: authenticate success, global_id 4532 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff048c8d640 10 monclient: _renew_subs 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff048c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff048c8d640 1 -- 172.21.15.153:0/3833242985 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff044138530 con 0x7ff0440b0860 2024-03-27T03:12:56.902 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff048c8d640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff048c8d640 10 monclient: _renew_subs 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff048c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff048c8d640 1 -- 172.21.15.153:0/3833242985 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0441df660 con 0x7ff0440b0860 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.904+0000 7ff048c8d640 1 librados: init done 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.906+0000 7ff048c8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.906+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff03c0020a0 con 0x7ff0440b0860 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.906+0000 7ff048c8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.906+0000 7ff048c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.906+0000 7ff048c8d640 1 -- 172.21.15.153:0/3833242985 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff044136f60 con 0x7ff0440b0860 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.906+0000 7ff032ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.906+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff03c004d80 con 0x7ff0440b0860 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.907+0000 7ff032ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.907+0000 7ff032ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.907+0000 7ff032ffd640 10 monclient: dump: 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:12:56.903 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:12:56.904 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:56.904 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:12:56.904 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:12:56.904 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:12:56.904 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:12:56.904 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:12:56.904 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:12:56.904 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:12:56.904 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.907+0000 7ff033fff640 4 set_mon_vals no callback set 2024-03-27T03:12:56.904 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.907+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7ff03c01ad90 con 0x7ff0440b0860 2024-03-27T03:12:56.905 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.912+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7ff03c084d40 con 0x7ff0440b0860 2024-03-27T03:12:56.905 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.912+0000 7ff032ffd640 10 client.4532.objecter ms_dispatch 0x7ff044000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:12:56.905 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.912+0000 7ff032ffd640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:12:56.906 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.912+0000 7ff032ffd640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:12:56.906 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.912+0000 7ff032ffd640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-27T03:12:56.906 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.913+0000 7ff04259c640 10 monclient: get_auth_request con 0x7ff02406dfb0 auth_method 0 2024-03-27T03:12:56.908 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.915+0000 7ff032ffd640 10 client.4532.objecter ms_handle_connect 0x7ff02406dfb0 2024-03-27T03:12:56.909 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.916+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff044136f60 con 0x7ff0440b0860 2024-03-27T03:12:56.909 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.916+0000 7ff032ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:12:56.910 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:56.916+0000 7ff032ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:12:57.233 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.240+0000 7ff048c8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:57.233 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.240+0000 7ff048c8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:57.233 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.240+0000 7ff048c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:12:57.233 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.240+0000 7ff048c8d640 1 -- 172.21.15.153:0/3833242985 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0441386c0 con 0x7ff0440b0860 2024-03-27T03:12:57.234 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.241+0000 7ff032ffd640 1 -- 172.21.15.153:0/3833242985 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7ff044136f60 con 0x7ff0440b0860 2024-03-27T03:12:57.234 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.241+0000 7ff032ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:12:57.234 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.241+0000 7ff032ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:12:57.235 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:12:57.235 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:12:57.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.245+0000 7ff048c8d640 10 librados: watch_flush enter 2024-03-27T03:12:57.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.245+0000 7ff048c8d640 10 librados: watch_flush exit 2024-03-27T03:12:57.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.245+0000 7ff048c8d640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-27T03:12:57.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.245+0000 7ff048c8d640 10 client.4532.objecter successfully canceled tick 2024-03-27T03:12:57.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.245+0000 7ff048c8d640 1 -- 172.21.15.153:0/3833242985 >> v1:172.21.15.153:6816/2637937781 conn(0x7ff02406dfb0 legacy=0x7ff024070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:57.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.245+0000 7ff048c8d640 10 monclient: shutdown 2024-03-27T03:12:57.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.245+0000 7ff048c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:12:57.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.245+0000 7ff048c8d640 1 -- 172.21.15.153:0/3833242985 >> v1:172.21.15.153:6790/0 conn(0x7ff0440b0860 legacy=0x7ff0441ded70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:12:57.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.247+0000 7ff048c8d640 1 -- 172.21.15.153:0/3833242985 shutdown_connections 2024-03-27T03:12:57.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.247+0000 7ff048c8d640 1 -- 172.21.15.153:0/3833242985 >> 172.21.15.153:0/3833242985 conn(0x7ff04406b660 msgr2=0x7ff04413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:12:57.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.247+0000 7ff048c8d640 1 -- 172.21.15.153:0/3833242985 shutdown_connections 2024-03-27T03:12:57.241 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.248+0000 7ff048c8d640 1 -- 172.21.15.153:0/3833242985 wait complete. 2024-03-27T03:12:57.241 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:12:57.248+0000 7ff048c8d640 1 librados: shutdown 2024-03-27T03:12:57.259 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:13:04.068 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:13:04.229 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.235+0000 7f86d6d62640 10 monclient: get_monmap_and_config 2024-03-27T03:13:04.229 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.235+0000 7f86d6d62640 10 monclient: build_initial_monmap 2024-03-27T03:13:04.229 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.235+0000 7f86d6d62640 10 monclient: monmap: 2024-03-27T03:13:04.229 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:04.229 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:04.230 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:13:04.236877+0000 2024-03-27T03:13:04.230 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:13:04.236877+0000 2024-03-27T03:13:04.230 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:04.230 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:04.230 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:04.230 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:13:04.230 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:13:04.230 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:04.230 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.236+0000 7f86d6d62640 1 Processor -- start 2024-03-27T03:13:04.232 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.237+0000 7f86d6d62640 1 -- start start 2024-03-27T03:13:04.232 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.237+0000 7f86d6d62640 10 monclient: init 2024-03-27T03:13:04.232 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.238+0000 7f86d6d62640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:04.232 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.238+0000 7f86d6d62640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:13:04.233 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.238+0000 7f86d6d62640 10 monclient(hunting): picked mon.noname-b con 0x7f86c80389e0 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:04.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.241+0000 7f86d6d62640 10 monclient(hunting): picked mon.noname-c con 0x7f86c8038dc0 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:04.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.242+0000 7f86d6d62640 10 monclient(hunting): picked mon.noname-a con 0x7f86c8030660 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:04.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.242+0000 7f86d6d62640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c80c54c0 con 0x7f86c8030660 2024-03-27T03:13:04.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.242+0000 7f86d6d62640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c8036430 con 0x7f86c8038dc0 2024-03-27T03:13:04.235 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.242+0000 7f86d6d62640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c8039840 con 0x7f86c80389e0 2024-03-27T03:13:04.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.242+0000 7f86d6d62640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:04.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.242+0000 7f86d6d62640 10 monclient(hunting): authenticate will time out at 1744.548180s 2024-03-27T03:13:04.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d6561640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f86c8030660 0x7f86c80d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:35138/0 (socket says 172.21.15.153:35138) 2024-03-27T03:13:04.236 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d6561640 1 -- 172.21.15.153:0/2482876495 learned_addr learned my addr 172.21.15.153:0/2482876495 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:04.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080393579 0 0) 0x7f86c80c54c0 con 0x7f86c8030660 2024-03-27T03:13:04.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d4d5e640 10 monclient(hunting): _negotiate 2024-03-27T03:13:04.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d4d5e640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:04.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d4d5e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:04.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d4d5e640 10 monclient(hunting): my global_id is 4470 2024-03-27T03:13:04.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86bc003520 con 0x7f86c8030660 2024-03-27T03:13:04.237 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439004567 0 0) 0x7f86c8039840 con 0x7f86c80389e0 2024-03-27T03:13:04.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d4d5e640 10 monclient(hunting): _negotiate 2024-03-27T03:13:04.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d4d5e640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:04.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d4d5e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:04.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d4d5e640 10 monclient(hunting): my global_id is 4535 2024-03-27T03:13:04.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.243+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c80c54c0 con 0x7f86c80389e0 2024-03-27T03:13:04.238 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.244+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248063704 0 0) 0x7f86c8036430 con 0x7f86c8038dc0 2024-03-27T03:13:04.239 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.244+0000 7f86d4d5e640 10 monclient(hunting): _negotiate 2024-03-27T03:13:04.239 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.244+0000 7f86d4d5e640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:04.239 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.244+0000 7f86d4d5e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:04.239 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.244+0000 7f86d4d5e640 10 monclient(hunting): my global_id is 4501 2024-03-27T03:13:04.239 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.244+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c8039840 con 0x7f86c8038dc0 2024-03-27T03:13:04.239 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2337408460 0 0) 0x7f86c80c54c0 con 0x7f86c80389e0 2024-03-27T03:13:04.239 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 >> v1:172.21.15.196:6789/0 conn(0x7f86c8038dc0 legacy=0x7f86c80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:04.239 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 >> v1:172.21.15.153:6789/0 conn(0x7f86c8030660 legacy=0x7f86c80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:04.239 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 10 monclient: _finish_hunting 0 2024-03-27T03:13:04.239 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 1 monclient: found mon.noname-b 2024-03-27T03:13:04.239 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:04.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:13:04.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c8035690 con 0x7f86c80389e0 2024-03-27T03:13:04.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 10 monclient: _finish_auth 0 2024-03-27T03:13:04.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 10 monclient: _check_auth_tickets 2024-03-27T03:13:04.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:04.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f86c0001610 con 0x7f86c80389e0 2024-03-27T03:13:04.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:04.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:13:04.240 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 10 monclient: dump: 2024-03-27T03:13:04.240 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:04.240 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:04.240 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:04.241 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:04.241 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:04.241 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:04.241 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:04.241 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:04.241 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:04.241 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:04.241 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86c0001770 con 0x7f86c80389e0 2024-03-27T03:13:04.241 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:04.241 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 1 -- 172.21.15.153:0/2482876495 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f86c0001b80 con 0x7f86c80389e0 2024-03-27T03:13:04.241 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:04.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:13:04.242 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d4d5e640 10 monclient: dump: 2024-03-27T03:13:04.242 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:04.242 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:04.242 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:04.242 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:04.242 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:04.242 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:04.242 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:04.242 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:04.243 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:04.243 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:04.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d6d62640 5 monclient: authenticate success, global_id 4535 2024-03-27T03:13:04.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d6d62640 10 monclient: get_monmap_and_config success 2024-03-27T03:13:04.243 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.245+0000 7f86d6d62640 4 set_mon_vals no callback set 2024-03-27T03:13:04.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.246+0000 7f86d6d62640 10 monclient: shutdown 2024-03-27T03:13:04.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.246+0000 7f86d6d62640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:04.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.246+0000 7f86d6d62640 1 -- 172.21.15.153:0/2482876495 >> v1:172.21.15.153:6790/0 conn(0x7f86c80389e0 legacy=0x7f86c8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:04.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.246+0000 7f86d6d62640 1 -- 172.21.15.153:0/2482876495 shutdown_connections 2024-03-27T03:13:04.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.246+0000 7f86d6d62640 1 -- 172.21.15.153:0/2482876495 >> 172.21.15.153:0/2482876495 conn(0x7f86c8005c00 msgr2=0x7f86c8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:04.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.246+0000 7f86d6d62640 1 -- 172.21.15.153:0/2482876495 shutdown_connections 2024-03-27T03:13:04.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.246+0000 7f86d6d62640 1 -- 172.21.15.153:0/2482876495 wait complete. 2024-03-27T03:13:04.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.247+0000 7f86d6d62640 10 monclient: build_initial_monmap 2024-03-27T03:13:04.248 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.247+0000 7f86d6d62640 10 monclient: monmap: 2024-03-27T03:13:04.248 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:04.248 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:13:04.248 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:13:04.249 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:13:04.249 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:04.249 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:04.249 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:04.249 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:13:04.249 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:13:04.249 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:04.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.248+0000 7f86d6d62640 1 librados: starting msgr at 2024-03-27T03:13:04.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.248+0000 7f86d6d62640 1 librados: starting objecter 2024-03-27T03:13:04.249 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.248+0000 7f86d6d62640 1 Processor -- start 2024-03-27T03:13:04.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.248+0000 7f86d6d62640 1 -- start start 2024-03-27T03:13:04.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.248+0000 7f86d6d62640 1 librados: setting wanted keys 2024-03-27T03:13:04.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.248+0000 7f86d6d62640 1 librados: calling monclient init 2024-03-27T03:13:04.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.248+0000 7f86d6d62640 10 monclient: init 2024-03-27T03:13:04.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.249+0000 7f86d6d62640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:04.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.249+0000 7f86d6d62640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:13:04.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.249+0000 7f86d6d62640 10 monclient(hunting): picked mon.noname-c con 0x7f86c8030660 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:04.250 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.249+0000 7f86d6d62640 10 monclient(hunting): picked mon.noname-a con 0x7f86c80389e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:04.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.249+0000 7f86d6d62640 10 monclient(hunting): picked mon.noname-b con 0x7f86c80c71b0 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:04.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.249+0000 7f86d6d62640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c80c8040 con 0x7f86c80389e0 2024-03-27T03:13:04.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.249+0000 7f86d6d62640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c802f790 con 0x7f86c80c71b0 2024-03-27T03:13:04.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.249+0000 7f86d6d62640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c802f920 con 0x7f86c8030660 2024-03-27T03:13:04.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.249+0000 7f86d6d62640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:04.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.249+0000 7f86d6d62640 10 monclient(hunting): authenticate will time out at 1744.555362s 2024-03-27T03:13:04.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.250+0000 7f86d555f640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f86c80389e0 0x7f86c815fbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45670/0 (socket says 172.21.15.153:45670) 2024-03-27T03:13:04.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.250+0000 7f86d555f640 1 -- 172.21.15.153:0/3143355922 learned_addr learned my addr 172.21.15.153:0/3143355922 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:04.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.251+0000 7f86c67fc640 10 client.?.objecter ms_handle_connect 0x7f86c80389e0 2024-03-27T03:13:04.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.251+0000 7f86c67fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:04.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.251+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544749975 0 0) 0x7f86c80c8040 con 0x7f86c80389e0 2024-03-27T03:13:04.251 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.251+0000 7f86c67fc640 10 monclient(hunting): _negotiate 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.251+0000 7f86c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.251+0000 7f86c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.251+0000 7f86c67fc640 10 monclient(hunting): my global_id is 4473 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.251+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b4004eb0 con 0x7f86c80389e0 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3120274616 0 0) 0x7f86b4004eb0 con 0x7f86c80389e0 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86c80c8040 con 0x7f86c80389e0 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 10 client.?.objecter ms_handle_connect 0x7f86c80c71b0 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f86cc001a20 con 0x7f86c80389e0 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 10 monclient(hunting): dump: 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:04.252 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 10 client.?.objecter ms_handle_connect 0x7f86c8030660 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472912916 0 0) 0x7f86c802f790 con 0x7f86c80c71b0 2024-03-27T03:13:04.253 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 10 monclient(hunting): _negotiate 2024-03-27T03:13:04.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:04.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:04.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 10 monclient(hunting): my global_id is 4504 2024-03-27T03:13:04.254 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.252+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b4004eb0 con 0x7f86c80c71b0 2024-03-27T03:13:04.258 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1620612849 0 0) 0x7f86c80c8040 con 0x7f86c80389e0 2024-03-27T03:13:04.258 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 >> v1:172.21.15.153:6790/0 conn(0x7f86c8030660 legacy=0x7f86c80c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:04.258 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 >> v1:172.21.15.196:6789/0 conn(0x7f86c80c71b0 legacy=0x7f86c8167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:04.258 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 10 monclient: _finish_hunting 0 2024-03-27T03:13:04.258 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 1 monclient: found mon.a 2024-03-27T03:13:04.258 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:04.258 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:13:04.258 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c8160300 con 0x7f86c80389e0 2024-03-27T03:13:04.258 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 10 monclient: _finish_auth 0 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 10 monclient: _check_auth_tickets 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86d6d62640 5 monclient: authenticate success, global_id 4473 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86d6d62640 10 monclient: _renew_subs 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86d6d62640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86d6d62640 1 -- 172.21.15.153:0/3143355922 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c8160550 con 0x7f86c80389e0 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86d6d62640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86d6d62640 10 monclient: _renew_subs 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86d6d62640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86d6d62640 1 -- 172.21.15.153:0/3143355922 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c80097b0 con 0x7f86c80389e0 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86d6d62640 1 librados: init done 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86cc0038c0 con 0x7f86c80389e0 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f86cc001d80 con 0x7f86c80389e0 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:04.259 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:13:04.260 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.253+0000 7f86c67fc640 10 monclient: dump: 2024-03-27T03:13:04.260 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:04.260 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:04.260 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:04.260 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:04.260 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:04.260 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:04.260 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:04.260 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:04.260 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:04.260 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.255+0000 7f86bbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.255+0000 7f86bbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.255+0000 7f86bbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.255+0000 7f86bbfff640 1 -- 172.21.15.153:0/3143355922 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b0005080 con 0x7f86c80389e0 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.255+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f86cc003cd0 con 0x7f86c80389e0 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.256+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f86cc084c50 con 0x7f86c80389e0 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.256+0000 7f86c67fc640 10 client.4473.objecter ms_dispatch 0x7f86d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.256+0000 7f86c67fc640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.256+0000 7f86c67fc640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.256+0000 7f86c67fc640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.256+0000 7f86c7fff640 4 set_mon_vals no callback set 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.256+0000 7f86d5d60640 10 monclient: get_auth_request con 0x7f86b407ed30 auth_method 0 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.257+0000 7f86c67fc640 10 client.4473.objecter ms_handle_connect 0x7f86b407ed30 2024-03-27T03:13:04.261 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.264+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86cc050530 con 0x7f86c80389e0 2024-03-27T03:13:04.262 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.265+0000 7f86c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:04.262 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.265+0000 7f86c67fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:13:04.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.595+0000 7f86bbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:04.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.595+0000 7f86bbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:04.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.595+0000 7f86bbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:13:04.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.595+0000 7f86bbfff640 1 -- 172.21.15.153:0/3143355922 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86b0005330 con 0x7f86c80389e0 2024-03-27T03:13:04.589 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.595+0000 7f86c67fc640 1 -- 172.21.15.153:0/3143355922 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f86cc052460 con 0x7f86c80389e0 2024-03-27T03:13:04.589 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.595+0000 7f86c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:04.589 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.595+0000 7f86c67fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:13:04.589 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:13:04.589 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:13:04.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.599+0000 7f86bbfff640 10 librados: watch_flush enter 2024-03-27T03:13:04.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.600+0000 7f86bbfff640 10 librados: watch_flush exit 2024-03-27T03:13:04.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.600+0000 7f86bbfff640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-27T03:13:04.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.600+0000 7f86bbfff640 10 client.4473.objecter successfully canceled tick 2024-03-27T03:13:04.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.600+0000 7f86bbfff640 1 -- 172.21.15.153:0/3143355922 >> v1:172.21.15.153:6816/2637937781 conn(0x7f86b407ed30 legacy=0x7f86b40811f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:04.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.600+0000 7f86bbfff640 10 monclient: shutdown 2024-03-27T03:13:04.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.600+0000 7f86bbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:04.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.600+0000 7f86bbfff640 1 -- 172.21.15.153:0/3143355922 >> v1:172.21.15.153:6789/0 conn(0x7f86c80389e0 legacy=0x7f86c815fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:04.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.601+0000 7f86bbfff640 1 -- 172.21.15.153:0/3143355922 shutdown_connections 2024-03-27T03:13:04.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.601+0000 7f86bbfff640 1 -- 172.21.15.153:0/3143355922 >> 172.21.15.153:0/3143355922 conn(0x7f86c8005c00 msgr2=0x7f86c8034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:04.594 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.601+0000 7f86bbfff640 1 -- 172.21.15.153:0/3143355922 shutdown_connections 2024-03-27T03:13:04.595 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.601+0000 7f86bbfff640 1 -- 172.21.15.153:0/3143355922 wait complete. 2024-03-27T03:13:04.595 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:04.602+0000 7f86bbfff640 1 librados: shutdown 2024-03-27T03:13:04.615 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:13:11.422 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:13:11.569 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.575+0000 7fdf52a55640 10 monclient: get_monmap_and_config 2024-03-27T03:13:11.569 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.576+0000 7fdf52a55640 10 monclient: build_initial_monmap 2024-03-27T03:13:11.569 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.576+0000 7fdf52a55640 10 monclient: monmap: 2024-03-27T03:13:11.569 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:11.569 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:11.569 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:13:11.576936+0000 2024-03-27T03:13:11.569 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:13:11.576936+0000 2024-03-27T03:13:11.569 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:11.570 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:11.570 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:11.570 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:13:11.570 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:13:11.570 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:11.578 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.577+0000 7fdf52a55640 1 Processor -- start 2024-03-27T03:13:11.578 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.577+0000 7fdf52a55640 1 -- start start 2024-03-27T03:13:11.578 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.577+0000 7fdf52a55640 10 monclient: init 2024-03-27T03:13:11.578 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.578+0000 7fdf52a55640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:11.578 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.578+0000 7fdf52a55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:13:11.579 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.578+0000 7fdf52a55640 10 monclient(hunting): picked mon.noname-b con 0x7fdf4c1422b0 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:11.579 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.578+0000 7fdf52a55640 10 monclient(hunting): picked mon.noname-c con 0x7fdf4c142690 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:11.579 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.578+0000 7fdf4bfff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fdf4c1422b0 0x7fdf4c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:58880/0 (socket says 172.21.15.153:58880) 2024-03-27T03:13:11.579 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.578+0000 7fdf52a55640 10 monclient(hunting): picked mon.noname-a con 0x7fdf4c0b17d0 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:11.579 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.578+0000 7fdf4bfff640 1 -- 172.21.15.153:0/3951043963 learned_addr learned my addr 172.21.15.153:0/3951043963 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:11.579 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.578+0000 7fdf52a55640 1 -- 172.21.15.153:0/3951043963 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c142c80 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.579 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.578+0000 7fdf52a55640 1 -- 172.21.15.153:0/3951043963 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c0aae40 con 0x7fdf4c142690 2024-03-27T03:13:11.579 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.578+0000 7fdf52a55640 1 -- 172.21.15.153:0/3951043963 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c13fbd0 con 0x7fdf4c1422b0 2024-03-27T03:13:11.579 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf52a55640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:11.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf52a55640 10 monclient(hunting): authenticate will time out at 1751.884619s 2024-03-27T03:13:11.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf4affd640 1 -- 172.21.15.153:0/3951043963 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937303246 0 0) 0x7fdf4c142c80 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf4affd640 10 monclient(hunting): _negotiate 2024-03-27T03:13:11.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf4affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:11.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:11.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf4affd640 10 monclient(hunting): my global_id is 4476 2024-03-27T03:13:11.580 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf4affd640 1 -- 172.21.15.153:0/3951043963 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf34003520 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.581 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf4affd640 1 -- 172.21.15.153:0/3951043963 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288273225 0 0) 0x7fdf4c13fbd0 con 0x7fdf4c1422b0 2024-03-27T03:13:11.581 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf4affd640 10 monclient(hunting): _negotiate 2024-03-27T03:13:11.581 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf4affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:11.581 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:11.581 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf4affd640 10 monclient(hunting): my global_id is 4541 2024-03-27T03:13:11.581 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.579+0000 7fdf4affd640 1 -- 172.21.15.153:0/3951043963 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf4c142c80 con 0x7fdf4c1422b0 2024-03-27T03:13:11.581 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 1 -- 172.21.15.153:0/3951043963 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1960926394 0 0) 0x7fdf34003520 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 1 -- 172.21.15.153:0/3951043963 >> v1:172.21.15.153:6790/0 conn(0x7fdf4c1422b0 legacy=0x7fdf4c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:11.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 1 -- 172.21.15.153:0/3951043963 >> v1:172.21.15.196:6789/0 conn(0x7fdf4c142690 legacy=0x7fdf4c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:13:11.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 10 monclient: _finish_hunting 0 2024-03-27T03:13:11.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 1 monclient: found mon.noname-a 2024-03-27T03:13:11.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:11.582 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:13:11.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 1 -- 172.21.15.153:0/3951043963 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c14e910 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 10 monclient: _finish_auth 0 2024-03-27T03:13:11.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 10 monclient: _check_auth_tickets 2024-03-27T03:13:11.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:11.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 1 -- 172.21.15.153:0/3951043963 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdf3c0015c0 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:11.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:13:11.583 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 10 monclient: dump: 2024-03-27T03:13:11.583 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:11.583 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf52a55640 5 monclient: authenticate success, global_id 4476 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf52a55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 1 -- 172.21.15.153:0/3951043963 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf3c001720 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf4affd640 1 -- 172.21.15.153:0/3951043963 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdf3c001b70 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.584 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf52a55640 10 monclient: get_monmap_and_config success 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.580+0000 7fdf52a55640 4 set_mon_vals no callback set 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.581+0000 7fdf4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.581+0000 7fdf4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.581+0000 7fdf4affd640 10 monclient: dump: 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.581+0000 7fdf52a55640 10 monclient: shutdown 2024-03-27T03:13:11.585 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.581+0000 7fdf52a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:11.586 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.581+0000 7fdf52a55640 1 -- 172.21.15.153:0/3951043963 >> v1:172.21.15.153:6789/0 conn(0x7fdf4c0b17d0 legacy=0x7fdf4c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:11.586 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.582+0000 7fdf52a55640 1 -- 172.21.15.153:0/3951043963 shutdown_connections 2024-03-27T03:13:11.586 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.582+0000 7fdf52a55640 1 -- 172.21.15.153:0/3951043963 >> 172.21.15.153:0/3951043963 conn(0x7fdf4c066a00 msgr2=0x7fdf4c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:11.586 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.582+0000 7fdf52a55640 1 -- 172.21.15.153:0/3951043963 shutdown_connections 2024-03-27T03:13:11.586 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.582+0000 7fdf52a55640 1 -- 172.21.15.153:0/3951043963 wait complete. 2024-03-27T03:13:11.586 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.583+0000 7fdf52a55640 10 monclient: build_initial_monmap 2024-03-27T03:13:11.586 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.583+0000 7fdf52a55640 10 monclient: monmap: 2024-03-27T03:13:11.586 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:11.586 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:13:11.586 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:13:11.586 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:13:11.586 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.583+0000 7fdf52a55640 1 librados: starting msgr at 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.583+0000 7fdf52a55640 1 librados: starting objecter 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.584+0000 7fdf52a55640 1 Processor -- start 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.584+0000 7fdf52a55640 1 -- start start 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.584+0000 7fdf52a55640 1 librados: setting wanted keys 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.584+0000 7fdf52a55640 1 librados: calling monclient init 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.584+0000 7fdf52a55640 10 monclient: init 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf52a55640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf52a55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:13:11.587 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf52a55640 10 monclient(hunting): picked mon.noname-c con 0x7fdf4c0b17d0 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:11.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf52a55640 10 monclient(hunting): picked mon.noname-b con 0x7fdf4c1422b0 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:11.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf52a55640 10 monclient(hunting): picked mon.noname-a con 0x7fdf4c142690 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:11.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf52a55640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c134920 con 0x7fdf4c142690 2024-03-27T03:13:11.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf52a55640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c134c00 con 0x7fdf4c1422b0 2024-03-27T03:13:11.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf52a55640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c130050 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf52a55640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:11.588 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf52a55640 10 monclient(hunting): authenticate will time out at 1751.890737s 2024-03-27T03:13:11.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf4b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdf4c1422b0 0x7fdf4c133a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:57982/0 (socket says 172.21.15.153:57982) 2024-03-27T03:13:11.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf4b7fe640 1 -- 172.21.15.153:0/3011931662 learned_addr learned my addr 172.21.15.153:0/3011931662 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:11.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.585+0000 7fdf48ff9640 10 client.?.objecter ms_handle_connect 0x7fdf4c0b17d0 2024-03-27T03:13:11.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:11.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 client.?.objecter ms_handle_connect 0x7fdf4c1422b0 2024-03-27T03:13:11.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:11.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 client.?.objecter ms_handle_connect 0x7fdf4c142690 2024-03-27T03:13:11.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:11.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319799484 0 0) 0x7fdf4c134920 con 0x7fdf4c142690 2024-03-27T03:13:11.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:13:11.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:11.600 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:11.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): my global_id is 4479 2024-03-27T03:13:11.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf38004eb0 con 0x7fdf4c142690 2024-03-27T03:13:11.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062022319 0 0) 0x7fdf4c134c00 con 0x7fdf4c1422b0 2024-03-27T03:13:11.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:13:11.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:11.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:11.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): my global_id is 4507 2024-03-27T03:13:11.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf4c134920 con 0x7fdf4c1422b0 2024-03-27T03:13:11.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397505343 0 0) 0x7fdf4c130050 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:13:11.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:11.601 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:11.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): my global_id is 4544 2024-03-27T03:13:11.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf4c134c00 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1516706051 0 0) 0x7fdf38004eb0 con 0x7fdf4c142690 2024-03-27T03:13:11.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf4c130050 con 0x7fdf4c142690 2024-03-27T03:13:11.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdf3c001a30 con 0x7fdf4c142690 2024-03-27T03:13:11.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:11.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:13:11.602 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): dump: 2024-03-27T03:13:11.602 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:11.602 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:11.602 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.586+0000 7fdf48ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2474681017 0 0) 0x7fdf4c134920 con 0x7fdf4c1422b0 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf38004eb0 con 0x7fdf4c1422b0 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdf40003f60 con 0x7fdf4c1422b0 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-27T03:13:11.603 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 10 monclient(hunting): dump: 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3765134323 0 0) 0x7fdf4c134c00 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf4c134920 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.604 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 445624868 0 0) 0x7fdf38004eb0 con 0x7fdf4c1422b0 2024-03-27T03:13:11.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 >> v1:172.21.15.153:6790/0 conn(0x7fdf4c0b17d0 legacy=0x7fdf4c0b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:11.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 >> v1:172.21.15.153:6789/0 conn(0x7fdf4c142690 legacy=0x7fdf4c1e1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:11.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:13:11.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 1 monclient: found mon.b 2024-03-27T03:13:11.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:11.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:11.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c134190 con 0x7fdf4c1422b0 2024-03-27T03:13:11.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 10 monclient: _finish_auth 0 2024-03-27T03:13:11.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:13:11.605 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.587+0000 7fdf48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:11.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf52a55640 5 monclient: authenticate success, global_id 4507 2024-03-27T03:13:11.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf52a55640 10 monclient: _renew_subs 2024-03-27T03:13:11.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf52a55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:11.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf52a55640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4c12e830 con 0x7fdf4c1422b0 2024-03-27T03:13:11.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf52a55640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:13:11.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf52a55640 10 monclient: _renew_subs 2024-03-27T03:13:11.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf52a55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:11.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf52a55640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4c130650 con 0x7fdf4c1422b0 2024-03-27T03:13:11.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf52a55640 1 librados: init done 2024-03-27T03:13:11.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf400040c0 con 0x7fdf4c1422b0 2024-03-27T03:13:11.606 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf48ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:11.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdf40001070 con 0x7fdf4c1422b0 2024-03-27T03:13:11.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:11.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf4a7fc640 4 set_mon_vals no callback set 2024-03-27T03:13:11.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf48ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:13:11.607 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.588+0000 7fdf48ff9640 10 monclient: dump: 2024-03-27T03:13:11.607 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:11.607 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:11.607 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:11.607 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:11.607 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:11.607 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:11.607 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.589+0000 7fdf52a55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.589+0000 7fdf52a55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.589+0000 7fdf52a55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.589+0000 7fdf52a55640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2c005080 con 0x7fdf4c1422b0 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.591+0000 7fdf4b7fe640 1 -- 172.21.15.153:0/3011931662 >> v1:172.21.15.196:6789/0 conn(0x7fdf4c1422b0 legacy=0x7fdf4c133a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.591+0000 7fdf4b7fe640 1 -- 172.21.15.153:0/3011931662 >> v1:172.21.15.196:6789/0 conn(0x7fdf4c1422b0 legacy=0x7fdf4c133a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.591+0000 7fdf4b7fe640 1 --1- 172.21.15.153:0/3011931662 >> v1:172.21.15.196:6789/0 conn(0x7fdf4c1422b0 0x7fdf4c133a80 :-1 s=OPENED pgs=951762 cs=1 l=1).handle_message read tag failed 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.591+0000 7fdf4b7fe640 1 --1- 172.21.15.153:0/3011931662 >> v1:172.21.15.196:6789/0 conn(0x7fdf4c1422b0 0x7fdf4c133a80 :-1 s=OPENED pgs=951762 cs=1 l=1).fault on lossy channel, failing 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.591+0000 7fdf48ff9640 10 monclient: ms_handle_reset current mon v1:172.21.15.196:6789/0 2024-03-27T03:13:11.608 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.591+0000 7fdf48ff9640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.591+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 >> v1:172.21.15.196:6789/0 conn(0x7fdf4c1422b0 legacy=0x7fdf4c133a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.591+0000 7fdf48ff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.591+0000 7fdf48ff9640 10 monclient(hunting): picked mon.c con 0x7fdf4c0b17d0 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.591+0000 7fdf48ff9640 10 monclient(hunting): picked mon.a con 0x7fdf4c142690 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 10 monclient(hunting): picked mon.b con 0x7fdf3800e320 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdf38004eb0 con 0x7fdf4c142690 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdf4c130050 con 0x7fdf3800e320 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdf4c134920 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 10 client.4507.objecter ms_handle_connect 0x7fdf4c0b17d0 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 10 client.4507.objecter resend_mon_ops 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 10 client.4507.objecter ms_handle_connect 0x7fdf4c142690 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 10 client.4507.objecter resend_mon_ops 2024-03-27T03:13:11.609 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856829365 0 0) 0x7fdf4c134920 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:13:11.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:11.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T03:13:11.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.153:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fdf38011f90 con 0x7fdf4c0b17d0 2024-03-27T03:13:11.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791590000 0 0) 0x7fdf38004eb0 con 0x7fdf4c142690 2024-03-27T03:13:11.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:13:11.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:11.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T03:13:11.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.592+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.153:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fdf4c134920 con 0x7fdf4c142690 2024-03-27T03:13:11.610 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 4182142704 0 0) 0x7fdf4c134920 con 0x7fdf4c142690 2024-03-27T03:13:11.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 >> v1:172.21.15.153:6790/0 conn(0x7fdf4c0b17d0 legacy=0x7fdf380061e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:11.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 >> v1:172.21.15.196:6789/0 conn(0x7fdf3800e320 legacy=0x7fdf38010710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:11.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:13:11.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 1 monclient: found mon.a 2024-03-27T03:13:11.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:11.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:13:11.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fdf4c134c00 con 0x7fdf4c142690 2024-03-27T03:13:11.611 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf38011f90 con 0x7fdf4c142690 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 10 monclient: _finish_auth 0 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 10 client.4507.objecter ms_handle_connect 0x7fdf3800e320 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 10 client.4507.objecter resend_mon_ops 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdf40053d20 con 0x7fdf4c142690 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 10 monclient: dump: 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:11.612 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:11.613 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:11.613 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:11.613 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:11.613 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:11.613 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:11.613 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:11.613 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:11.613 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:11.613 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf40054650 con 0x7fdf4c142690 2024-03-27T03:13:11.613 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf48ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:11.614 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.593+0000 7fdf49ffb640 4 set_mon_vals no callback set 2024-03-27T03:13:11.614 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.595+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fdf400011d0 con 0x7fdf4c142690 2024-03-27T03:13:11.614 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.596+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.0 v1:172.21.15.153:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fdf40084880 con 0x7fdf4c142690 2024-03-27T03:13:11.614 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.596+0000 7fdf48ff9640 10 client.4507.objecter ms_dispatch 0x7fdf4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:13:11.614 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.596+0000 7fdf48ff9640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:13:11.614 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.596+0000 7fdf48ff9640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:13:11.614 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.596+0000 7fdf4bfff640 10 monclient: get_auth_request con 0x7fdf4c1422b0 auth_method 0 2024-03-27T03:13:11.614 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.596+0000 7fdf48ff9640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-27T03:13:11.614 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.596+0000 7fdf48ff9640 10 client.4507.objecter ms_handle_connect 0x7fdf4c1422b0 2024-03-27T03:13:11.614 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.602+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf40075950 con 0x7fdf4c142690 2024-03-27T03:13:11.614 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.602+0000 7fdf48ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:11.614 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.602+0000 7fdf48ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:13:11.970 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.976+0000 7fdf52a55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:11.970 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.976+0000 7fdf52a55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:11.970 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.976+0000 7fdf52a55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:13:11.970 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.976+0000 7fdf52a55640 1 -- 172.21.15.153:0/3011931662 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf2c005330 con 0x7fdf4c142690 2024-03-27T03:13:11.970 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.977+0000 7fdf48ff9640 1 -- 172.21.15.153:0/3011931662 <== mon.0 v1:172.21.15.153:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fdf40079010 con 0x7fdf4c142690 2024-03-27T03:13:11.971 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.977+0000 7fdf48ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:11.971 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.977+0000 7fdf48ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:13:11.971 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:13:11.971 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:13:11.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.981+0000 7fdf52a55640 10 librados: watch_flush enter 2024-03-27T03:13:11.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.981+0000 7fdf52a55640 10 librados: watch_flush exit 2024-03-27T03:13:11.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.982+0000 7fdf52a55640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-27T03:13:11.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.982+0000 7fdf52a55640 10 client.4507.objecter successfully canceled tick 2024-03-27T03:13:11.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.982+0000 7fdf52a55640 1 -- 172.21.15.153:0/3011931662 >> v1:172.21.15.153:6816/2637937781 conn(0x7fdf4c1422b0 legacy=0x7fdf3807b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:11.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.982+0000 7fdf52a55640 10 monclient: shutdown 2024-03-27T03:13:11.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.982+0000 7fdf52a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:11.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.982+0000 7fdf52a55640 1 -- 172.21.15.153:0/3011931662 >> v1:172.21.15.153:6789/0 conn(0x7fdf4c142690 legacy=0x7fdf3800cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:11.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.982+0000 7fdf52a55640 1 -- 172.21.15.153:0/3011931662 shutdown_connections 2024-03-27T03:13:11.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.982+0000 7fdf52a55640 1 -- 172.21.15.153:0/3011931662 >> 172.21.15.153:0/3011931662 conn(0x7fdf4c066a00 msgr2=0x7fdf4c128ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:11.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.982+0000 7fdf52a55640 1 -- 172.21.15.153:0/3011931662 shutdown_connections 2024-03-27T03:13:11.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.982+0000 7fdf52a55640 1 -- 172.21.15.153:0/3011931662 wait complete. 2024-03-27T03:13:11.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:11.982+0000 7fdf52a55640 1 librados: shutdown 2024-03-27T03:13:11.988 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:13:18.793 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:13:18.952 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.959+0000 7f75ca80f640 10 monclient: get_monmap_and_config 2024-03-27T03:13:18.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.959+0000 7f75ca80f640 10 monclient: build_initial_monmap 2024-03-27T03:13:18.953 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.959+0000 7f75ca80f640 10 monclient: monmap: 2024-03-27T03:13:18.953 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:18.953 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:18.953 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:13:18.960585+0000 2024-03-27T03:13:18.953 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:13:18.960585+0000 2024-03-27T03:13:18.953 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:18.953 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:18.953 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:18.953 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:13:18.953 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:13:18.953 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:18.954 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.961+0000 7f75ca80f640 1 Processor -- start 2024-03-27T03:13:18.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.961+0000 7f75ca80f640 1 -- start start 2024-03-27T03:13:18.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.961+0000 7f75ca80f640 10 monclient: init 2024-03-27T03:13:18.958 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.962+0000 7f75ca80f640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:18.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.962+0000 7f75ca80f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:13:18.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.962+0000 7f75ca80f640 10 monclient(hunting): picked mon.noname-c con 0x7f75c40b0770 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:18.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.962+0000 7f75ca80f640 10 monclient(hunting): picked mon.noname-b con 0x7f75c414a700 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:18.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.962+0000 7f75ca80f640 10 monclient(hunting): picked mon.noname-a con 0x7f75c40ac5a0 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:18.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.962+0000 7f75ca80f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c40aba70 con 0x7f75c40ac5a0 2024-03-27T03:13:18.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.962+0000 7f75ca80f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c40aecb0 con 0x7f75c40b0770 2024-03-27T03:13:18.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.962+0000 7f75ca80f640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c40af150 con 0x7f75c414a700 2024-03-27T03:13:18.959 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.962+0000 7f75ca80f640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:18.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.962+0000 7f75ca80f640 10 monclient(hunting): authenticate will time out at 1759.268353s 2024-03-27T03:13:18.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c3fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f75c40b0770 0x7f75c4149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:35440/0 (socket says 172.21.15.153:35440) 2024-03-27T03:13:18.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c3fff640 1 -- 172.21.15.153:0/10980751 learned_addr learned my addr 172.21.15.153:0/10980751 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:18.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c2ffd640 1 -- 172.21.15.153:0/10980751 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511159080 0 0) 0x7f75c40aecb0 con 0x7f75c40b0770 2024-03-27T03:13:18.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c2ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:13:18.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:18.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:18.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c2ffd640 10 monclient(hunting): my global_id is 4510 2024-03-27T03:13:18.960 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c2ffd640 1 -- 172.21.15.153:0/10980751 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a8003520 con 0x7f75c40b0770 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c2ffd640 1 -- 172.21.15.153:0/10980751 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959262594 0 0) 0x7f75c40aba70 con 0x7f75c40ac5a0 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c2ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c2ffd640 10 monclient(hunting): my global_id is 4482 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.963+0000 7f75c2ffd640 1 -- 172.21.15.153:0/10980751 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c40aecb0 con 0x7f75c40ac5a0 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 1 -- 172.21.15.153:0/10980751 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1869022304 0 0) 0x7f75a8003520 con 0x7f75c40b0770 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 1 -- 172.21.15.153:0/10980751 >> v1:172.21.15.153:6790/0 conn(0x7f75c414a700 legacy=0x7f75c40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 1 -- 172.21.15.153:0/10980751 >> v1:172.21.15.153:6789/0 conn(0x7f75c40ac5a0 legacy=0x7f75c4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 1 monclient: found mon.noname-c 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 1 -- 172.21.15.153:0/10980751 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c40ac9d0 con 0x7f75c40b0770 2024-03-27T03:13:18.961 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 10 monclient: _finish_auth 0 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75ca80f640 5 monclient: authenticate success, global_id 4510 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75ca80f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 1 -- 172.21.15.153:0/10980751 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f75bc001a30 con 0x7f75c40b0770 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 10 monclient: dump: 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:18.962 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75ca80f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 1 -- 172.21.15.153:0/10980751 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75bc001c50 con 0x7f75c40b0770 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75ca80f640 10 monclient: get_monmap_and_config success 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75ca80f640 4 set_mon_vals no callback set 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 1 -- 172.21.15.153:0/10980751 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f75bc001fe0 con 0x7f75c40b0770 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.964+0000 7f75c2ffd640 10 monclient: dump: 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:18.963 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.965+0000 7f75ca80f640 10 monclient: shutdown 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.965+0000 7f75ca80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.965+0000 7f75ca80f640 1 -- 172.21.15.153:0/10980751 >> v1:172.21.15.196:6789/0 conn(0x7f75c40b0770 legacy=0x7f75c4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.965+0000 7f75ca80f640 1 -- 172.21.15.153:0/10980751 shutdown_connections 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.965+0000 7f75ca80f640 1 -- 172.21.15.153:0/10980751 >> 172.21.15.153:0/10980751 conn(0x7f75c406b660 msgr2=0x7f75c406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.965+0000 7f75ca80f640 1 -- 172.21.15.153:0/10980751 shutdown_connections 2024-03-27T03:13:18.964 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.965+0000 7f75ca80f640 1 -- 172.21.15.153:0/10980751 wait complete. 2024-03-27T03:13:18.970 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.966+0000 7f75ca80f640 10 monclient: build_initial_monmap 2024-03-27T03:13:18.970 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.966+0000 7f75ca80f640 10 monclient: monmap: 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.966+0000 7f75ca80f640 1 librados: starting msgr at 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.966+0000 7f75ca80f640 1 librados: starting objecter 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.966+0000 7f75ca80f640 1 Processor -- start 2024-03-27T03:13:18.971 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.967+0000 7f75ca80f640 1 -- start start 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.967+0000 7f75ca80f640 1 librados: setting wanted keys 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.967+0000 7f75ca80f640 1 librados: calling monclient init 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.967+0000 7f75ca80f640 10 monclient: init 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.967+0000 7f75ca80f640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.967+0000 7f75ca80f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75ca80f640 10 monclient(hunting): picked mon.noname-c con 0x7f75c40ac5a0 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75ca80f640 10 monclient(hunting): picked mon.noname-a con 0x7f75c40b0770 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75ca80f640 10 monclient(hunting): picked mon.noname-b con 0x7f75c414a700 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75ca80f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c41df930 con 0x7f75c40b0770 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75ca80f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c41dfdc0 con 0x7f75c414a700 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75ca80f640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c4124340 con 0x7f75c40ac5a0 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75ca80f640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75c3fff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f75c40ac5a0 0x7f75c4123c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:46080/0 (socket says 172.21.15.153:46080) 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75c3fff640 1 -- 172.21.15.153:0/829117222 learned_addr learned my addr 172.21.15.153:0/829117222 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:18.972 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75ca80f640 10 monclient(hunting): authenticate will time out at 1759.273866s 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75c0ff9640 10 client.?.objecter ms_handle_connect 0x7f75c40ac5a0 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693389597 0 0) 0x7f75c4124340 con 0x7f75c40ac5a0 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75c0ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75c0ff9640 10 monclient(hunting): my global_id is 4547 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75ac004eb0 con 0x7f75c40ac5a0 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75c0ff9640 10 client.?.objecter ms_handle_connect 0x7f75c414a700 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.968+0000 7f75c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301196266 0 0) 0x7f75c41dfdc0 con 0x7f75c414a700 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): my global_id is 4513 2024-03-27T03:13:18.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c4124340 con 0x7f75c414a700 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 client.?.objecter ms_handle_connect 0x7f75c40b0770 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3650808621 0 0) 0x7f75ac004eb0 con 0x7f75c40ac5a0 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75ac005490 con 0x7f75c40ac5a0 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f75bc001970 con 0x7f75c40ac5a0 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): dump: 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:18.974 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1565127583 0 0) 0x7f75c4124340 con 0x7f75c414a700 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75ac004eb0 con 0x7f75c414a700 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f75b40018f0 con 0x7f75c414a700 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): dump: 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:18.975 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996833187 0 0) 0x7f75c41df930 con 0x7f75c40b0770 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 10 monclient(hunting): my global_id is 4485 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.969+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c4124340 con 0x7f75c40b0770 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1671987218 0 0) 0x7f75ac005490 con 0x7f75c40ac5a0 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 >> v1:172.21.15.196:6789/0 conn(0x7f75c414a700 legacy=0x7f75c4126c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 >> v1:172.21.15.153:6789/0 conn(0x7f75c40b0770 legacy=0x7f75c4126510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75c0ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75c0ff9640 1 monclient: found mon.c 2024-03-27T03:13:18.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75c0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c41244d0 con 0x7f75c40ac5a0 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75c0ff9640 10 monclient: _finish_auth 0 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75c0ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75ca80f640 5 monclient: authenticate success, global_id 4547 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75ca80f640 10 monclient: _renew_subs 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75ca80f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75ca80f640 1 -- 172.21.15.153:0/829117222 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c4145310 con 0x7f75c40ac5a0 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75ca80f640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75ca80f640 10 monclient: _renew_subs 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75ca80f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75ca80f640 1 -- 172.21.15.153:0/829117222 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c41df480 con 0x7f75c40ac5a0 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.970+0000 7f75ca80f640 1 librados: init done 2024-03-27T03:13:18.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.971+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75bc0015d0 con 0x7f75c40ac5a0 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.971+0000 7f75c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.971+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f75bc004fb0 con 0x7f75c40ac5a0 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.971+0000 7f75c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.971+0000 7f75c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.971+0000 7f75c0ff9640 10 monclient: dump: 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:18.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.971+0000 7f75c27fc640 4 set_mon_vals no callback set 2024-03-27T03:13:18.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.972+0000 7f75ca80f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:18.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.972+0000 7f75ca80f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:18.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.972+0000 7f75ca80f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:18.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.972+0000 7f75ca80f640 1 -- 172.21.15.153:0/829117222 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a0005080 con 0x7f75c40ac5a0 2024-03-27T03:13:18.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.973+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f75bc01af80 con 0x7f75c40ac5a0 2024-03-27T03:13:18.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.973+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f75bc084ef0 con 0x7f75c40ac5a0 2024-03-27T03:13:18.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.973+0000 7f75c0ff9640 10 client.4547.objecter ms_dispatch 0x7f75c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:13:18.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.973+0000 7f75c0ff9640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:13:18.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.973+0000 7f75c0ff9640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:13:18.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.973+0000 7f75c0ff9640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-27T03:13:18.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.974+0000 7f75c37fe640 10 monclient: get_auth_request con 0x7f75ac06e170 auth_method 0 2024-03-27T03:13:18.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.974+0000 7f75c0ff9640 10 client.4547.objecter ms_handle_connect 0x7f75ac06e170 2024-03-27T03:13:18.980 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.981+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75bc050600 con 0x7f75c40ac5a0 2024-03-27T03:13:18.980 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.981+0000 7f75c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:18.980 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:18.981+0000 7f75c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:13:19.362 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.369+0000 7f75ca80f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:19.363 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.369+0000 7f75ca80f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:19.363 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.369+0000 7f75ca80f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:19.363 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.369+0000 7f75ca80f640 1 -- 172.21.15.153:0/829117222 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75a0005330 con 0x7f75c40ac5a0 2024-03-27T03:13:19.364 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.372+0000 7f75c0ff9640 1 -- 172.21.15.153:0/829117222 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f75bc052700 con 0x7f75c40ac5a0 2024-03-27T03:13:19.365 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.372+0000 7f75c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:19.365 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.372+0000 7f75c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:13:19.365 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:13:19.365 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:13:19.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.375+0000 7f75ca80f640 10 librados: watch_flush enter 2024-03-27T03:13:19.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.375+0000 7f75ca80f640 10 librados: watch_flush exit 2024-03-27T03:13:19.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.375+0000 7f75ca80f640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-27T03:13:19.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.375+0000 7f75ca80f640 10 client.4547.objecter successfully canceled tick 2024-03-27T03:13:19.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.375+0000 7f75ca80f640 1 -- 172.21.15.153:0/829117222 >> v1:172.21.15.153:6816/2637937781 conn(0x7f75ac06e170 legacy=0x7f75ac070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:19.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.375+0000 7f75ca80f640 10 monclient: shutdown 2024-03-27T03:13:19.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.375+0000 7f75ca80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:19.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.375+0000 7f75ca80f640 1 -- 172.21.15.153:0/829117222 >> v1:172.21.15.153:6790/0 conn(0x7f75c40ac5a0 legacy=0x7f75c4123c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:19.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.376+0000 7f75ca80f640 1 -- 172.21.15.153:0/829117222 shutdown_connections 2024-03-27T03:13:19.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.376+0000 7f75ca80f640 1 -- 172.21.15.153:0/829117222 >> 172.21.15.153:0/829117222 conn(0x7f75c406b660 msgr2=0x7f75c40b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:19.370 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.376+0000 7f75ca80f640 1 -- 172.21.15.153:0/829117222 shutdown_connections 2024-03-27T03:13:19.370 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.376+0000 7f75ca80f640 1 -- 172.21.15.153:0/829117222 wait complete. 2024-03-27T03:13:19.370 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:19.377+0000 7f75ca80f640 1 librados: shutdown 2024-03-27T03:13:19.383 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:13:26.190 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:13:26.329 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.334+0000 7f278a70e640 10 monclient: get_monmap_and_config 2024-03-27T03:13:26.329 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.334+0000 7f278a70e640 10 monclient: build_initial_monmap 2024-03-27T03:13:26.329 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.334+0000 7f278a70e640 10 monclient: monmap: 2024-03-27T03:13:26.329 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:26.329 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:26.329 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:13:26.336817+0000 2024-03-27T03:13:26.329 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:13:26.336817+0000 2024-03-27T03:13:26.329 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:26.329 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:26.329 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:26.329 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:13:26.330 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:13:26.330 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:26.333 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 1 Processor -- start 2024-03-27T03:13:26.333 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 1 -- start start 2024-03-27T03:13:26.333 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 10 monclient: init 2024-03-27T03:13:26.333 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:26.333 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:13:26.334 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 10 monclient(hunting): picked mon.noname-c con 0x7f27841422b0 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:26.334 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 10 monclient(hunting): picked mon.noname-b con 0x7f2784142690 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:26.334 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 10 monclient(hunting): picked mon.noname-a con 0x7f27840b17d0 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:26.334 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2784142c80 con 0x7f27840b17d0 2024-03-27T03:13:26.334 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27840aae40 con 0x7f27841422b0 2024-03-27T03:13:26.334 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278413fbd0 con 0x7f2784142690 2024-03-27T03:13:26.334 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:26.334 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.336+0000 7f278a70e640 10 monclient(hunting): authenticate will time out at 1766.643514s 2024-03-27T03:13:26.334 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.337+0000 7f277b7fe640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f2784142690 0x7f27840b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:35368/0 (socket says 172.21.15.153:35368) 2024-03-27T03:13:26.334 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.337+0000 7f277b7fe640 1 -- 172.21.15.153:0/1296948516 learned_addr learned my addr 172.21.15.153:0/1296948516 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:26.335 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.337+0000 7f27837fe640 1 -- 172.21.15.153:0/1296948516 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770771929 0 0) 0x7f278413fbd0 con 0x7f2784142690 2024-03-27T03:13:26.335 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.337+0000 7f27837fe640 10 monclient(hunting): _negotiate 2024-03-27T03:13:26.335 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.337+0000 7f27837fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:26.335 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.337+0000 7f27837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:26.335 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.337+0000 7f27837fe640 10 monclient(hunting): my global_id is 4550 2024-03-27T03:13:26.335 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.337+0000 7f27837fe640 1 -- 172.21.15.153:0/1296948516 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2770003520 con 0x7f2784142690 2024-03-27T03:13:26.335 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.337+0000 7f27837fe640 1 -- 172.21.15.153:0/1296948516 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3273891209 0 0) 0x7f2770003520 con 0x7f2784142690 2024-03-27T03:13:26.335 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.337+0000 7f27837fe640 1 -- 172.21.15.153:0/1296948516 >> v1:172.21.15.196:6789/0 conn(0x7f27841422b0 legacy=0x7f278413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:13:26.335 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.337+0000 7f27837fe640 1 -- 172.21.15.153:0/1296948516 >> v1:172.21.15.153:6789/0 conn(0x7f27840b17d0 legacy=0x7f27840b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:13:26.337 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f27837fe640 10 monclient: _finish_hunting 0 2024-03-27T03:13:26.337 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f27837fe640 1 monclient: found mon.noname-b 2024-03-27T03:13:26.337 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f27837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:26.337 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f27837fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:13:26.338 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f27837fe640 1 -- 172.21.15.153:0/1296948516 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2784151940 con 0x7f2784142690 2024-03-27T03:13:26.338 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f27837fe640 10 monclient: _finish_auth 0 2024-03-27T03:13:26.338 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f27837fe640 10 monclient: _check_auth_tickets 2024-03-27T03:13:26.338 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f27837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:26.338 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f27837fe640 1 -- 172.21.15.153:0/1296948516 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f276c001690 con 0x7f2784142690 2024-03-27T03:13:26.338 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f27837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:26.338 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f27837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:13:26.338 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f27837fe640 10 monclient: dump: 2024-03-27T03:13:26.338 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:26.338 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:26.338 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:26.338 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:26.339 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:26.339 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:26.339 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:26.339 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:26.339 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:26.339 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:26.339 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f278a70e640 5 monclient: authenticate success, global_id 4550 2024-03-27T03:13:26.339 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.340+0000 7f278a70e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:13:26.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.344+0000 7f27837fe640 1 -- 172.21.15.153:0/1296948516 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f276c0017f0 con 0x7f2784142690 2024-03-27T03:13:26.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.344+0000 7f27837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:26.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.344+0000 7f278a70e640 10 monclient: get_monmap_and_config success 2024-03-27T03:13:26.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.344+0000 7f278a70e640 4 set_mon_vals no callback set 2024-03-27T03:13:26.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.344+0000 7f27837fe640 1 -- 172.21.15.153:0/1296948516 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f276c001c40 con 0x7f2784142690 2024-03-27T03:13:26.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.344+0000 7f27837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:26.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.344+0000 7f27837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:13:26.345 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.344+0000 7f27837fe640 10 monclient: dump: 2024-03-27T03:13:26.345 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:26.345 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:26.345 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:26.345 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.345+0000 7f278a70e640 10 monclient: shutdown 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.345+0000 7f278a70e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.345+0000 7f278a70e640 1 -- 172.21.15.153:0/1296948516 >> v1:172.21.15.153:6790/0 conn(0x7f2784142690 legacy=0x7f27840b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.345+0000 7f278a70e640 1 -- 172.21.15.153:0/1296948516 shutdown_connections 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.345+0000 7f278a70e640 1 -- 172.21.15.153:0/1296948516 >> 172.21.15.153:0/1296948516 conn(0x7f2784066a00 msgr2=0x7f2784066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.345+0000 7f278a70e640 1 -- 172.21.15.153:0/1296948516 shutdown_connections 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.345+0000 7f278a70e640 1 -- 172.21.15.153:0/1296948516 wait complete. 2024-03-27T03:13:26.346 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.346+0000 7f278a70e640 10 monclient: build_initial_monmap 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.346+0000 7f278a70e640 10 monclient: monmap: 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.347+0000 7f278a70e640 1 librados: starting msgr at 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.347+0000 7f278a70e640 1 librados: starting objecter 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.347+0000 7f278a70e640 1 Processor -- start 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.347+0000 7f278a70e640 1 -- start start 2024-03-27T03:13:26.347 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.347+0000 7f278a70e640 1 librados: setting wanted keys 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.347+0000 7f278a70e640 1 librados: calling monclient init 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.347+0000 7f278a70e640 10 monclient: init 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.348+0000 7f278a70e640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.348+0000 7f278a70e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.348+0000 7f278a70e640 10 monclient(hunting): picked mon.noname-c con 0x7f27840b17d0 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.348+0000 7f278a70e640 10 monclient(hunting): picked mon.noname-b con 0x7f27841422b0 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.348+0000 7f2783fff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f27840b17d0 0x7f27840aeef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:42952/0 (socket says 172.21.15.153:42952) 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.348+0000 7f277b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f27841422b0 0x7f27841370f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:33274/0 (socket says 172.21.15.153:33274) 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f2783fff640 1 -- 172.21.15.153:0/3798723073 learned_addr learned my addr 172.21.15.153:0/3798723073 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f278a70e640 10 monclient(hunting): picked mon.noname-a con 0x7f2784142690 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27840ac2a0 con 0x7f2784142690 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27840af600 con 0x7f27841422b0 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27840b01e0 con 0x7f27840b17d0 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f278a70e640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:26.348 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f278a70e640 10 monclient(hunting): authenticate will time out at 1766.655941s 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f27817fa640 10 client.?.objecter ms_handle_connect 0x7f27840b17d0 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f27817fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f27817fa640 10 client.?.objecter ms_handle_connect 0x7f27841422b0 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f27817fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900068703 0 0) 0x7f27840b01e0 con 0x7f27840b17d0 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f27817fa640 10 monclient(hunting): _negotiate 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f27817fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f27817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f27817fa640 10 monclient(hunting): my global_id is 4553 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.349+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2764004eb0 con 0x7f27840b17d0 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.350+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3205164071 0 0) 0x7f2764004eb0 con 0x7f27840b17d0 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.350+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27840b01e0 con 0x7f27840b17d0 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.350+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2774001960 con 0x7f27840b17d0 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.350+0000 7f27817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:26.349 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.350+0000 7f27817fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.350+0000 7f27817fa640 10 monclient(hunting): dump: 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.350+0000 7f27817fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.350+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3188402319 0 0) 0x7f27840b01e0 con 0x7f27840b17d0 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.350+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 >> v1:172.21.15.196:6789/0 conn(0x7f27841422b0 legacy=0x7f27841370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:26.350 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.350+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 >> v1:172.21.15.153:6789/0 conn(0x7f2784142690 legacy=0x7f27841dad00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:13:26.352 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 10 monclient: _finish_hunting 0 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 1 monclient: found mon.c 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27840b1070 con 0x7f27840b17d0 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 10 monclient: _finish_auth 0 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 10 monclient: _check_auth_tickets 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f278a70e640 5 monclient: authenticate success, global_id 4553 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f278a70e640 10 monclient: _renew_subs 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f278a70e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27840b1340 con 0x7f27840b17d0 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2774001c40 con 0x7f27840b17d0 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:26.353 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2774003a40 con 0x7f27840b17d0 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f27817fa640 10 monclient: dump: 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:26.354 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.352+0000 7f2782ffd640 4 set_mon_vals no callback set 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.353+0000 7f278a70e640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.353+0000 7f278a70e640 10 monclient: _renew_subs 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.353+0000 7f278a70e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.353+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2784137800 con 0x7f27840b17d0 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.353+0000 7f278a70e640 1 librados: init done 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.354+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f2774003c20 con 0x7f27840b17d0 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.355+0000 7f278a70e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.355+0000 7f278a70e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.355+0000 7f278a70e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.355+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27841daa20 con 0x7f27840b17d0 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.355+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f2774084bb0 con 0x7f27840b17d0 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.355+0000 7f27817fa640 10 client.4553.objecter ms_dispatch 0x7f2784000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.355+0000 7f27817fa640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.355+0000 7f27817fa640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:13:26.355 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.355+0000 7f277b7fe640 10 monclient: get_auth_request con 0x7f276406dc70 auth_method 0 2024-03-27T03:13:26.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.355+0000 7f27817fa640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-27T03:13:26.356 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.356+0000 7f27817fa640 10 client.4553.objecter ms_handle_connect 0x7f276406dc70 2024-03-27T03:13:26.357 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.363+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2774050490 con 0x7f27840b17d0 2024-03-27T03:13:26.357 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.363+0000 7f27817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:26.357 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.363+0000 7f27817fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:13:26.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.680+0000 7f278a70e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:26.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.680+0000 7f278a70e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:26.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.680+0000 7f278a70e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:26.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.680+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27840b0010 con 0x7f27840b17d0 2024-03-27T03:13:26.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.681+0000 7f27817fa640 1 -- 172.21.15.153:0/3798723073 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f27740523c0 con 0x7f27840b17d0 2024-03-27T03:13:26.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.681+0000 7f27817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:26.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.681+0000 7f27817fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:13:26.676 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:13:26.676 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:13:26.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 10 librados: watch_flush enter 2024-03-27T03:13:26.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 10 librados: watch_flush exit 2024-03-27T03:13:26.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-27T03:13:26.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 10 client.4553.objecter successfully canceled tick 2024-03-27T03:13:26.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 >> v1:172.21.15.153:6816/2637937781 conn(0x7f276406dc70 legacy=0x7f2764070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:26.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 10 monclient: shutdown 2024-03-27T03:13:26.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:26.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 >> v1:172.21.15.153:6790/0 conn(0x7f27840b17d0 legacy=0x7f27840aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:26.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 shutdown_connections 2024-03-27T03:13:26.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 >> 172.21.15.153:0/3798723073 conn(0x7f2784066a00 msgr2=0x7f278413ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:26.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 shutdown_connections 2024-03-27T03:13:26.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 1 -- 172.21.15.153:0/3798723073 wait complete. 2024-03-27T03:13:26.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:26.685+0000 7f278a70e640 1 librados: shutdown 2024-03-27T03:13:26.692 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:13:33.499 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:13:33.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.658+0000 7f10f91e8640 10 monclient: get_monmap_and_config 2024-03-27T03:13:33.652 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.658+0000 7f10f91e8640 10 monclient: build_initial_monmap 2024-03-27T03:13:33.653 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.658+0000 7f10f91e8640 10 monclient: monmap: 2024-03-27T03:13:33.653 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:33.653 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:33.653 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:13:33.659522+0000 2024-03-27T03:13:33.653 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:13:33.659522+0000 2024-03-27T03:13:33.653 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:33.653 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:33.653 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:33.653 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:13:33.653 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:13:33.653 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:33.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.662+0000 7f10f91e8640 1 Processor -- start 2024-03-27T03:13:33.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.662+0000 7f10f91e8640 1 -- start start 2024-03-27T03:13:33.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.662+0000 7f10f91e8640 10 monclient: init 2024-03-27T03:13:33.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.663+0000 7f10f91e8640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:33.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.663+0000 7f10f91e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:13:33.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.663+0000 7f10f91e8640 10 monclient(hunting): picked mon.noname-c con 0x7f10ec034850 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:33.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.663+0000 7f10f91e8640 10 monclient(hunting): picked mon.noname-a con 0x7f10ec0c84e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:33.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.663+0000 7f10f91e8640 10 monclient(hunting): picked mon.noname-b con 0x7f10ec0c51e0 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:33.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.663+0000 7f10f91e8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec029170 con 0x7f10ec0c84e0 2024-03-27T03:13:33.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.663+0000 7f10f91e8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec0293e0 con 0x7f10ec034850 2024-03-27T03:13:33.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.663+0000 7f10f91e8640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec032960 con 0x7f10ec0c51e0 2024-03-27T03:13:33.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.663+0000 7f10f91e8640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:33.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.663+0000 7f10f91e8640 10 monclient(hunting): authenticate will time out at 1773.968931s 2024-03-27T03:13:33.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.671+0000 7f10f37fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f10ec0c84e0 0x7f10ec0c4ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:43957/0 (socket says 172.21.15.153:43957) 2024-03-27T03:13:33.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.671+0000 7f10f37fe640 1 -- 172.21.15.153:0/443853240 learned_addr learned my addr 172.21.15.153:0/443853240 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:33.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.671+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941757732 0 0) 0x7f10ec032960 con 0x7f10ec0c51e0 2024-03-27T03:13:33.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.671+0000 7f10f2ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:13:33.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.671+0000 7f10f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:33.665 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.671+0000 7f10f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:33.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.671+0000 7f10f2ffd640 10 monclient(hunting): my global_id is 4556 2024-03-27T03:13:33.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.671+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e4003520 con 0x7f10ec0c51e0 2024-03-27T03:13:33.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.672+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408887348 0 0) 0x7f10ec0293e0 con 0x7f10ec034850 2024-03-27T03:13:33.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.672+0000 7f10f2ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:13:33.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.672+0000 7f10f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:33.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.672+0000 7f10f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:33.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.672+0000 7f10f2ffd640 10 monclient(hunting): my global_id is 4519 2024-03-27T03:13:33.666 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.672+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e4003840 con 0x7f10ec034850 2024-03-27T03:13:33.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646156680 0 0) 0x7f10ec029170 con 0x7f10ec0c84e0 2024-03-27T03:13:33.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:13:33.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:33.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:33.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 10 monclient(hunting): my global_id is 4488 2024-03-27T03:13:33.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ec0293e0 con 0x7f10ec0c84e0 2024-03-27T03:13:33.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 712314065 0 0) 0x7f10e4003520 con 0x7f10ec0c51e0 2024-03-27T03:13:33.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 >> v1:172.21.15.196:6789/0 conn(0x7f10ec034850 legacy=0x7f10ec0c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 >> v1:172.21.15.153:6789/0 conn(0x7f10ec0c84e0 legacy=0x7f10ec0c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 1 monclient: found mon.noname-b 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec0c78b0 con 0x7f10ec0c51e0 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 10 monclient: _finish_auth 0 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f10e0001670 con 0x7f10ec0c51e0 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.674+0000 7f10f2ffd640 10 monclient: dump: 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:33.668 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:33.669 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:33.669 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:33.669 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:33.669 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:33.669 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:33.669 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:33.669 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.675+0000 7f10f91e8640 5 monclient: authenticate success, global_id 4556 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.675+0000 7f10f91e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.675+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10e0001850 con 0x7f10ec0c51e0 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.675+0000 7f10f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.675+0000 7f10f91e8640 10 monclient: get_monmap_and_config success 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.675+0000 7f10f91e8640 4 set_mon_vals no callback set 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.675+0000 7f10f2ffd640 1 -- 172.21.15.153:0/443853240 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f10e0001670 con 0x7f10ec0c51e0 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.675+0000 7f10f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.676+0000 7f10f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.676+0000 7f10f2ffd640 10 monclient: dump: 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:33.670 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:33.671 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:33.671 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:33.671 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:33.671 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:33.671 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:33.671 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:33.671 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:33.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.676+0000 7f10f91e8640 10 monclient: shutdown 2024-03-27T03:13:33.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.676+0000 7f10f91e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:33.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.676+0000 7f10f91e8640 1 -- 172.21.15.153:0/443853240 >> v1:172.21.15.153:6790/0 conn(0x7f10ec0c51e0 legacy=0x7f10ec0d3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:33.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.677+0000 7f10f91e8640 1 -- 172.21.15.153:0/443853240 shutdown_connections 2024-03-27T03:13:33.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.677+0000 7f10f91e8640 1 -- 172.21.15.153:0/443853240 >> 172.21.15.153:0/443853240 conn(0x7f10ec0058f0 msgr2=0x7f10ec005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:33.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.679+0000 7f10f91e8640 1 -- 172.21.15.153:0/443853240 shutdown_connections 2024-03-27T03:13:33.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.679+0000 7f10f91e8640 1 -- 172.21.15.153:0/443853240 wait complete. 2024-03-27T03:13:33.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.679+0000 7f10f91e8640 10 monclient: build_initial_monmap 2024-03-27T03:13:33.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.679+0000 7f10f91e8640 10 monclient: monmap: 2024-03-27T03:13:33.673 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:33.673 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:13:33.673 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:13:33.673 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:13:33.673 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:33.673 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:33.673 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:33.673 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:13:33.673 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:13:33.673 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:33.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.680+0000 7f10f91e8640 1 librados: starting msgr at 2024-03-27T03:13:33.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.680+0000 7f10f91e8640 1 librados: starting objecter 2024-03-27T03:13:33.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.680+0000 7f10f91e8640 1 Processor -- start 2024-03-27T03:13:33.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.680+0000 7f10f91e8640 1 -- start start 2024-03-27T03:13:33.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.680+0000 7f10f91e8640 1 librados: setting wanted keys 2024-03-27T03:13:33.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.680+0000 7f10f91e8640 1 librados: calling monclient init 2024-03-27T03:13:33.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.680+0000 7f10f91e8640 10 monclient: init 2024-03-27T03:13:33.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.681+0000 7f10f91e8640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:33.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.681+0000 7f10f91e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:13:33.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.681+0000 7f10f91e8640 10 monclient(hunting): picked mon.noname-a con 0x7f10ec034850 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:33.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.681+0000 7f10f91e8640 10 monclient(hunting): picked mon.noname-c con 0x7f10ec0c51e0 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:33.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.681+0000 7f10f91e8640 10 monclient(hunting): picked mon.noname-b con 0x7f10ec0c84e0 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:33.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.681+0000 7f10f91e8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec0cb2f0 con 0x7f10ec034850 2024-03-27T03:13:33.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.681+0000 7f10f91e8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec0ca730 con 0x7f10ec0c84e0 2024-03-27T03:13:33.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.681+0000 7f10f91e8640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec0cb5d0 con 0x7f10ec0c51e0 2024-03-27T03:13:33.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.681+0000 7f10f91e8640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:33.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.681+0000 7f10f91e8640 10 monclient(hunting): authenticate will time out at 1773.987373s 2024-03-27T03:13:33.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.685+0000 7f10f37fe640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f10ec0c51e0 0x7f10ec0c1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:57203/0 (socket says 172.21.15.153:57203) 2024-03-27T03:13:33.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.685+0000 7f10f37fe640 1 -- 172.21.15.153:0/179163654 learned_addr learned my addr 172.21.15.153:0/179163654 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:33.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.685+0000 7f10f0ff9640 10 client.?.objecter ms_handle_connect 0x7f10ec0c51e0 2024-03-27T03:13:33.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.685+0000 7f10f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:33.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850943408 0 0) 0x7f10ec0cb5d0 con 0x7f10ec0c51e0 2024-03-27T03:13:33.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:13:33.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 10 monclient(hunting): my global_id is 4559 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10d8004eb0 con 0x7f10ec0c51e0 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2069454385 0 0) 0x7f10d8004eb0 con 0x7f10ec0c51e0 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10ec0cb5d0 con 0x7f10ec0c51e0 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f10f404fdd0 con 0x7f10ec0c51e0 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 10 monclient(hunting): dump: 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:33.681 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:33.682 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:33.682 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:33.682 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:33.682 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:33.682 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:33.682 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.686+0000 7f10f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:13:33.682 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.687+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1271731482 0 0) 0x7f10ec0cb5d0 con 0x7f10ec0c51e0 2024-03-27T03:13:33.682 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.687+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 >> v1:172.21.15.196:6789/0 conn(0x7f10ec0c84e0 legacy=0x7f10ec16b710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:13:33.683 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.689+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 >> v1:172.21.15.153:6789/0 conn(0x7f10ec034850 legacy=0x7f10ec0c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:33.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.689+0000 7f10f0ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:13:33.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.689+0000 7f10f0ff9640 1 monclient: found mon.c 2024-03-27T03:13:33.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.689+0000 7f10f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:33.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.689+0000 7f10f0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:33.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.689+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec0ca210 con 0x7f10ec0c51e0 2024-03-27T03:13:33.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.689+0000 7f10f0ff9640 10 monclient: _finish_auth 0 2024-03-27T03:13:33.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.689+0000 7f10f0ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:13:33.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.689+0000 7f10f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:33.685 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.690+0000 7f10f91e8640 5 monclient: authenticate success, global_id 4559 2024-03-27T03:13:33.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.690+0000 7f10f91e8640 10 monclient: _renew_subs 2024-03-27T03:13:33.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.690+0000 7f10f91e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:33.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.690+0000 7f10f91e8640 1 -- 172.21.15.153:0/179163654 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ec0c9c10 con 0x7f10ec0c51e0 2024-03-27T03:13:33.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.690+0000 7f10f91e8640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:13:33.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.690+0000 7f10f91e8640 10 monclient: _renew_subs 2024-03-27T03:13:33.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.690+0000 7f10f91e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:33.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.690+0000 7f10f91e8640 1 -- 172.21.15.153:0/179163654 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ec0c2550 con 0x7f10ec0c51e0 2024-03-27T03:13:33.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.690+0000 7f10f91e8640 1 librados: init done 2024-03-27T03:13:33.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.692+0000 7f10f91e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:33.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.692+0000 7f10f91e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:33.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.692+0000 7f10f91e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:33.686 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.692+0000 7f10f91e8640 1 -- 172.21.15.153:0/179163654 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ec0ca400 con 0x7f10ec0c51e0 2024-03-27T03:13:33.687 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.693+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10f404f400 con 0x7f10ec0c51e0 2024-03-27T03:13:33.687 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.693+0000 7f10f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:33.687 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.693+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f10f4075710 con 0x7f10ec0c51e0 2024-03-27T03:13:33.687 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.693+0000 7f10f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:33.687 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.693+0000 7f10f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:13:33.687 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.693+0000 7f10f0ff9640 10 monclient: dump: 2024-03-27T03:13:33.687 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:33.687 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:33.687 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:33.687 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:33.687 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:33.687 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:33.688 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:33.688 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:33.688 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:33.688 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:33.688 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.693+0000 7f10f1ffb640 4 set_mon_vals no callback set 2024-03-27T03:13:33.689 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.695+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f10f4067bf0 con 0x7f10ec0c51e0 2024-03-27T03:13:33.690 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.696+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f10f40ff6c0 con 0x7f10ec0c51e0 2024-03-27T03:13:33.690 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.696+0000 7f10f0ff9640 10 client.4559.objecter ms_dispatch 0x7f10f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:13:33.690 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.696+0000 7f10f0ff9640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:13:33.690 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.696+0000 7f10f0ff9640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:13:33.690 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.696+0000 7f10f0ff9640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-27T03:13:33.690 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.696+0000 7f10f3fff640 10 monclient: get_auth_request con 0x7f10d806dc20 auth_method 0 2024-03-27T03:13:33.690 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.697+0000 7f10f0ff9640 10 client.4559.objecter ms_handle_connect 0x7f10d806dc20 2024-03-27T03:13:33.695 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.702+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10f40cafa0 con 0x7f10ec0c51e0 2024-03-27T03:13:33.696 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.703+0000 7f10f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:33.696 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:33.703+0000 7f10f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:13:34.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.031+0000 7f10f91e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:34.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.031+0000 7f10f91e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:34.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.031+0000 7f10f91e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:34.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.031+0000 7f10f91e8640 1 -- 172.21.15.153:0/179163654 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10ec004cd0 con 0x7f10ec0c51e0 2024-03-27T03:13:34.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.032+0000 7f10f0ff9640 1 -- 172.21.15.153:0/179163654 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f10f40cced0 con 0x7f10ec0c51e0 2024-03-27T03:13:34.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.032+0000 7f10f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:34.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.032+0000 7f10f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:13:34.026 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:13:34.026 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:13:34.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.036+0000 7f10f91e8640 10 librados: watch_flush enter 2024-03-27T03:13:34.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.036+0000 7f10f91e8640 10 librados: watch_flush exit 2024-03-27T03:13:34.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.036+0000 7f10f91e8640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-27T03:13:34.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.036+0000 7f10f91e8640 10 client.4559.objecter successfully canceled tick 2024-03-27T03:13:34.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.036+0000 7f10f91e8640 1 -- 172.21.15.153:0/179163654 >> v1:172.21.15.153:6816/2637937781 conn(0x7f10d806dc20 legacy=0x7f10d80700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:34.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.036+0000 7f10f91e8640 10 monclient: shutdown 2024-03-27T03:13:34.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.036+0000 7f10f91e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:34.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.036+0000 7f10f91e8640 1 -- 172.21.15.153:0/179163654 >> v1:172.21.15.153:6790/0 conn(0x7f10ec0c51e0 legacy=0x7f10ec0c1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:34.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.037+0000 7f10f91e8640 1 -- 172.21.15.153:0/179163654 shutdown_connections 2024-03-27T03:13:34.030 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.037+0000 7f10f91e8640 1 -- 172.21.15.153:0/179163654 >> 172.21.15.153:0/179163654 conn(0x7f10ec0058f0 msgr2=0x7f10ec0050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:34.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.039+0000 7f10f91e8640 1 -- 172.21.15.153:0/179163654 shutdown_connections 2024-03-27T03:13:34.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.039+0000 7f10f91e8640 1 -- 172.21.15.153:0/179163654 wait complete. 2024-03-27T03:13:34.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:34.040+0000 7f10f91e8640 1 librados: shutdown 2024-03-27T03:13:34.053 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:13:40.861 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:13:41.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.018+0000 7f5a89cf4640 10 monclient: get_monmap_and_config 2024-03-27T03:13:41.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.018+0000 7f5a89cf4640 10 monclient: build_initial_monmap 2024-03-27T03:13:41.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.018+0000 7f5a89cf4640 10 monclient: monmap: 2024-03-27T03:13:41.011 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:41.011 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:41.011 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:13:41.019134+0000 2024-03-27T03:13:41.011 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:13:41.019134+0000 2024-03-27T03:13:41.011 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:41.012 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:41.012 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:41.012 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:13:41.012 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:13:41.012 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:41.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.019+0000 7f5a89cf4640 1 Processor -- start 2024-03-27T03:13:41.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.019+0000 7f5a89cf4640 1 -- start start 2024-03-27T03:13:41.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.019+0000 7f5a89cf4640 10 monclient: init 2024-03-27T03:13:41.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.020+0000 7f5a89cf4640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:41.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.020+0000 7f5a89cf4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:13:41.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.020+0000 7f5a89cf4640 10 monclient(hunting): picked mon.noname-c con 0x7f5a8406ee40 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:41.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.020+0000 7f5a89cf4640 10 monclient(hunting): picked mon.noname-b con 0x7f5a840aeda0 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:41.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.020+0000 7f5a89cf4640 10 monclient(hunting): picked mon.noname-a con 0x7f5a8413c470 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:41.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.020+0000 7f5a89cf4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8406fdb0 con 0x7f5a8413c470 2024-03-27T03:13:41.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.020+0000 7f5a89cf4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a84071090 con 0x7f5a8406ee40 2024-03-27T03:13:41.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.020+0000 7f5a89cf4640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a84071370 con 0x7f5a840aeda0 2024-03-27T03:13:41.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.020+0000 7f5a89cf4640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:41.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.020+0000 7f5a89cf4640 10 monclient(hunting): authenticate will time out at 1781.325869s 2024-03-27T03:13:41.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.023+0000 7f5a83fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5a8413c470 0x7f5a8414a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36815/0 (socket says 172.21.15.153:36815) 2024-03-27T03:13:41.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.023+0000 7f5a83fff640 1 -- 172.21.15.153:0/2367012793 learned_addr learned my addr 172.21.15.153:0/2367012793 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:41.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.024+0000 7f5a827fc640 1 -- 172.21.15.153:0/2367012793 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185924224 0 0) 0x7f5a8406fdb0 con 0x7f5a8413c470 2024-03-27T03:13:41.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.024+0000 7f5a827fc640 10 monclient(hunting): _negotiate 2024-03-27T03:13:41.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.024+0000 7f5a827fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:41.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.024+0000 7f5a827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:41.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.024+0000 7f5a827fc640 10 monclient(hunting): my global_id is 4491 2024-03-27T03:13:41.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.024+0000 7f5a827fc640 1 -- 172.21.15.153:0/2367012793 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c003520 con 0x7f5a8413c470 2024-03-27T03:13:41.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 1 -- 172.21.15.153:0/2367012793 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3858660630 0 0) 0x7f5a6c003520 con 0x7f5a8413c470 2024-03-27T03:13:41.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 1 -- 172.21.15.153:0/2367012793 >> v1:172.21.15.153:6790/0 conn(0x7f5a840aeda0 legacy=0x7f5a84071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 1 -- 172.21.15.153:0/2367012793 >> v1:172.21.15.196:6789/0 conn(0x7f5a8406ee40 legacy=0x7f5a8413bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 10 monclient: _finish_hunting 0 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 1 monclient: found mon.noname-a 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 1 -- 172.21.15.153:0/2367012793 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a84070220 con 0x7f5a8413c470 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 10 monclient: _finish_auth 0 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 10 monclient: _check_auth_tickets 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 1 -- 172.21.15.153:0/2367012793 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f5a7c001a10 con 0x7f5a8413c470 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 10 monclient: dump: 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:41.023 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a89cf4640 5 monclient: authenticate success, global_id 4491 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a89cf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 1 -- 172.21.15.153:0/2367012793 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a7c001be0 con 0x7f5a8413c470 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 1 -- 172.21.15.153:0/2367012793 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f5a7c002030 con 0x7f5a8413c470 2024-03-27T03:13:41.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a827fc640 10 monclient: dump: 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a89cf4640 10 monclient: get_monmap_and_config success 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.027+0000 7f5a89cf4640 4 set_mon_vals no callback set 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.028+0000 7f5a89cf4640 10 monclient: shutdown 2024-03-27T03:13:41.025 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.028+0000 7f5a89cf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:41.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.028+0000 7f5a89cf4640 1 -- 172.21.15.153:0/2367012793 >> v1:172.21.15.153:6789/0 conn(0x7f5a8413c470 legacy=0x7f5a8414a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:41.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.028+0000 7f5a89cf4640 1 -- 172.21.15.153:0/2367012793 shutdown_connections 2024-03-27T03:13:41.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.028+0000 7f5a89cf4640 1 -- 172.21.15.153:0/2367012793 >> 172.21.15.153:0/2367012793 conn(0x7f5a84067470 msgr2=0x7f5a840b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:41.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.028+0000 7f5a89cf4640 1 -- 172.21.15.153:0/2367012793 shutdown_connections 2024-03-27T03:13:41.026 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.028+0000 7f5a89cf4640 1 -- 172.21.15.153:0/2367012793 wait complete. 2024-03-27T03:13:41.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.029+0000 7f5a89cf4640 10 monclient: build_initial_monmap 2024-03-27T03:13:41.031 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.029+0000 7f5a89cf4640 10 monclient: monmap: 2024-03-27T03:13:41.031 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:41.031 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:13:41.031 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:13:41.031 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:13:41.031 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:41.031 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:41.032 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:41.032 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:13:41.032 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:13:41.032 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:41.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.030+0000 7f5a89cf4640 1 librados: starting msgr at 2024-03-27T03:13:41.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.030+0000 7f5a89cf4640 1 librados: starting objecter 2024-03-27T03:13:41.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.030+0000 7f5a89cf4640 1 Processor -- start 2024-03-27T03:13:41.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.030+0000 7f5a89cf4640 1 -- start start 2024-03-27T03:13:41.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.030+0000 7f5a89cf4640 1 librados: setting wanted keys 2024-03-27T03:13:41.032 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.030+0000 7f5a89cf4640 1 librados: calling monclient init 2024-03-27T03:13:41.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.030+0000 7f5a89cf4640 10 monclient: init 2024-03-27T03:13:41.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.031+0000 7f5a89cf4640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:41.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.031+0000 7f5a89cf4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:13:41.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.031+0000 7f5a89cf4640 10 monclient(hunting): picked mon.noname-a con 0x7f5a8406ee40 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:41.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.031+0000 7f5a89cf4640 10 monclient(hunting): picked mon.noname-b con 0x7f5a840aeda0 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:41.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.031+0000 7f5a89cf4640 10 monclient(hunting): picked mon.noname-c con 0x7f5a8413c470 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:41.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.031+0000 7f5a89cf4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a841e18b0 con 0x7f5a8406ee40 2024-03-27T03:13:41.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.031+0000 7f5a89cf4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a84141ac0 con 0x7f5a840aeda0 2024-03-27T03:13:41.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.031+0000 7f5a89cf4640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a84140880 con 0x7f5a8413c470 2024-03-27T03:13:41.033 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.031+0000 7f5a89cf4640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:41.034 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.031+0000 7f5a89cf4640 10 monclient(hunting): authenticate will time out at 1781.337044s 2024-03-27T03:13:41.034 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.037+0000 7f5a83fff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f5a8413c470 0x7f5a841e3cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:55983/0 (socket says 172.21.15.153:55983) 2024-03-27T03:13:41.034 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.037+0000 7f5a83fff640 1 -- 172.21.15.153:0/1204520485 learned_addr learned my addr 172.21.15.153:0/1204520485 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:41.034 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.037+0000 7f5a6bfff640 10 client.?.objecter ms_handle_connect 0x7f5a8413c470 2024-03-27T03:13:41.034 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.037+0000 7f5a6bfff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:41.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.041+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383369396 0 0) 0x7f5a84140880 con 0x7f5a8413c470 2024-03-27T03:13:41.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.041+0000 7f5a6bfff640 10 monclient(hunting): _negotiate 2024-03-27T03:13:41.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.041+0000 7f5a6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:41.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.041+0000 7f5a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:41.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.041+0000 7f5a6bfff640 10 monclient(hunting): my global_id is 4565 2024-03-27T03:13:41.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.041+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a70004eb0 con 0x7f5a8413c470 2024-03-27T03:13:41.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.042+0000 7f5a6bfff640 10 client.?.objecter ms_handle_connect 0x7f5a840aeda0 2024-03-27T03:13:41.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.042+0000 7f5a6bfff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:41.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.042+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 713210986 0 0) 0x7f5a70004eb0 con 0x7f5a8413c470 2024-03-27T03:13:41.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.042+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a84140880 con 0x7f5a8413c470 2024-03-27T03:13:41.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.042+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f5a7c001ef0 con 0x7f5a8413c470 2024-03-27T03:13:41.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.042+0000 7f5a6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:41.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.042+0000 7f5a6bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:13:41.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.042+0000 7f5a6bfff640 10 monclient(hunting): dump: 2024-03-27T03:13:41.041 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:41.041 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:41.041 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:41.041 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:41.041 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:41.041 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:41.042 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:41.042 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:41.042 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:41.042 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:41.042 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.042+0000 7f5a6bfff640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:13:41.042 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.042+0000 7f5a6bfff640 10 client.?.objecter ms_handle_connect 0x7f5a8406ee40 2024-03-27T03:13:41.042 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.042+0000 7f5a6bfff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:41.042 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000991725 0 0) 0x7f5a84141ac0 con 0x7f5a840aeda0 2024-03-27T03:13:41.042 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 10 monclient(hunting): _negotiate 2024-03-27T03:13:41.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:41.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:41.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 10 monclient(hunting): my global_id is 4522 2024-03-27T03:13:41.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a70004eb0 con 0x7f5a840aeda0 2024-03-27T03:13:41.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186679979 0 0) 0x7f5a841e18b0 con 0x7f5a8406ee40 2024-03-27T03:13:41.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 10 monclient(hunting): _negotiate 2024-03-27T03:13:41.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:41.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:41.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 10 monclient(hunting): my global_id is 4494 2024-03-27T03:13:41.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a70005810 con 0x7f5a8406ee40 2024-03-27T03:13:41.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1724443437 0 0) 0x7f5a84140880 con 0x7f5a8413c470 2024-03-27T03:13:41.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 >> v1:172.21.15.196:6789/0 conn(0x7f5a840aeda0 legacy=0x7f5a840ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:41.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 >> v1:172.21.15.153:6789/0 conn(0x7f5a8406ee40 legacy=0x7f5a841411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:41.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 10 monclient: _finish_hunting 0 2024-03-27T03:13:41.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 1 monclient: found mon.c 2024-03-27T03:13:41.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:41.045 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:41.051 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a84140a10 con 0x7f5a8413c470 2024-03-27T03:13:41.051 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 10 monclient: _finish_auth 0 2024-03-27T03:13:41.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 10 monclient: _check_auth_tickets 2024-03-27T03:13:41.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:41.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a89cf4640 5 monclient: authenticate success, global_id 4565 2024-03-27T03:13:41.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a89cf4640 10 monclient: _renew_subs 2024-03-27T03:13:41.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a89cf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:41.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.043+0000 7f5a89cf4640 1 -- 172.21.15.153:0/1204520485 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a84136390 con 0x7f5a8413c470 2024-03-27T03:13:41.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.044+0000 7f5a89cf4640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:13:41.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.044+0000 7f5a89cf4640 10 monclient: _renew_subs 2024-03-27T03:13:41.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.044+0000 7f5a89cf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:41.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.044+0000 7f5a89cf4640 1 -- 172.21.15.153:0/1204520485 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a840adf80 con 0x7f5a8413c470 2024-03-27T03:13:41.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.044+0000 7f5a89cf4640 1 librados: init done 2024-03-27T03:13:41.052 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.048+0000 7f5a89cf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:41.053 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.048+0000 7f5a89cf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:41.053 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.048+0000 7f5a89cf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:41.053 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.048+0000 7f5a89cf4640 1 -- 172.21.15.153:0/1204520485 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a84075f10 con 0x7f5a8413c470 2024-03-27T03:13:41.053 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.048+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a7c002050 con 0x7f5a8413c470 2024-03-27T03:13:41.053 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.048+0000 7f5a6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:41.053 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.048+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f5a7c004fb0 con 0x7f5a8413c470 2024-03-27T03:13:41.053 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.048+0000 7f5a6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:41.053 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.048+0000 7f5a6bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:13:41.053 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.048+0000 7f5a6bfff640 10 monclient: dump: 2024-03-27T03:13:41.054 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:41.054 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:41.054 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:41.054 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:41.054 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:41.054 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:41.054 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:41.054 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:41.055 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:41.055 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:41.055 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.049+0000 7f5a81ffb640 4 set_mon_vals no callback set 2024-03-27T03:13:41.055 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.050+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f5a7c01af80 con 0x7f5a8413c470 2024-03-27T03:13:41.055 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.051+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f5a7c054ce0 con 0x7f5a8413c470 2024-03-27T03:13:41.055 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.051+0000 7f5a6bfff640 10 client.4565.objecter ms_dispatch 0x7f5a84000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:13:41.055 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.051+0000 7f5a6bfff640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:13:41.055 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.051+0000 7f5a6bfff640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:13:41.055 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.051+0000 7f5a837fe640 10 monclient: get_auth_request con 0x7f5a7006e070 auth_method 0 2024-03-27T03:13:41.056 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.051+0000 7f5a6bfff640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-27T03:13:41.056 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.051+0000 7f5a6bfff640 10 client.4565.objecter ms_handle_connect 0x7f5a7006e070 2024-03-27T03:13:41.056 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.061+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a7c0507f0 con 0x7f5a8413c470 2024-03-27T03:13:41.056 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.061+0000 7f5a6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:41.056 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.061+0000 7f5a6bfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:13:41.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.389+0000 7f5a89cf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:41.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.389+0000 7f5a89cf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:41.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.389+0000 7f5a89cf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:13:41.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.389+0000 7f5a89cf4640 1 -- 172.21.15.153:0/1204520485 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a84074ed0 con 0x7f5a8413c470 2024-03-27T03:13:41.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.391+0000 7f5a6bfff640 1 -- 172.21.15.153:0/1204520485 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f5a7c052720 con 0x7f5a8413c470 2024-03-27T03:13:41.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.391+0000 7f5a6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:41.385 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.391+0000 7f5a6bfff640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:13:41.385 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:13:41.385 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:13:41.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.395+0000 7f5a89cf4640 10 librados: watch_flush enter 2024-03-27T03:13:41.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.395+0000 7f5a89cf4640 10 librados: watch_flush exit 2024-03-27T03:13:41.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.395+0000 7f5a89cf4640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-27T03:13:41.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.395+0000 7f5a89cf4640 10 client.4565.objecter successfully canceled tick 2024-03-27T03:13:41.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.395+0000 7f5a89cf4640 1 -- 172.21.15.153:0/1204520485 >> v1:172.21.15.153:6816/2637937781 conn(0x7f5a7006e070 legacy=0x7f5a70070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:41.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.395+0000 7f5a89cf4640 10 monclient: shutdown 2024-03-27T03:13:41.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.395+0000 7f5a89cf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:41.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.395+0000 7f5a89cf4640 1 -- 172.21.15.153:0/1204520485 >> v1:172.21.15.153:6790/0 conn(0x7f5a8413c470 legacy=0x7f5a841e3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:41.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.396+0000 7f5a89cf4640 1 -- 172.21.15.153:0/1204520485 shutdown_connections 2024-03-27T03:13:41.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.400+0000 7f5a89cf4640 1 -- 172.21.15.153:0/1204520485 >> 172.21.15.153:0/1204520485 conn(0x7f5a84067470 msgr2=0x7f5a8413bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:41.393 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.400+0000 7f5a89cf4640 1 -- 172.21.15.153:0/1204520485 shutdown_connections 2024-03-27T03:13:41.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.401+0000 7f5a89cf4640 1 -- 172.21.15.153:0/1204520485 wait complete. 2024-03-27T03:13:41.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:41.401+0000 7f5a89cf4640 1 librados: shutdown 2024-03-27T03:13:41.414 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:13:48.220 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:13:48.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.375+0000 7fdcaacd6640 10 monclient: get_monmap_and_config 2024-03-27T03:13:48.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.375+0000 7fdcaacd6640 10 monclient: build_initial_monmap 2024-03-27T03:13:48.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.376+0000 7fdcaacd6640 10 monclient: monmap: 2024-03-27T03:13:48.369 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:48.369 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:48.369 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:13:48.376933+0000 2024-03-27T03:13:48.369 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:13:48.376933+0000 2024-03-27T03:13:48.369 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:48.369 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:48.369 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:48.369 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:13:48.370 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:13:48.370 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:48.370 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.377+0000 7fdcaacd6640 1 Processor -- start 2024-03-27T03:13:48.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.377+0000 7fdcaacd6640 1 -- start start 2024-03-27T03:13:48.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.377+0000 7fdcaacd6640 10 monclient: init 2024-03-27T03:13:48.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.378+0000 7fdcaacd6640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:48.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.378+0000 7fdcaacd6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:13:48.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.378+0000 7fdcaacd6640 10 monclient(hunting): picked mon.noname-a con 0x7fdc9c036a60 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:48.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.378+0000 7fdcaacd6640 10 monclient(hunting): picked mon.noname-b con 0x7fdc9c0c5de0 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:48.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.378+0000 7fdcaacd6640 10 monclient(hunting): picked mon.noname-c con 0x7fdc9c0c3070 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:48.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.378+0000 7fdcaacd6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c02cc60 con 0x7fdc9c036a60 2024-03-27T03:13:48.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.378+0000 7fdcaacd6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c02e040 con 0x7fdc9c0c3070 2024-03-27T03:13:48.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.378+0000 7fdcaacd6640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c02e320 con 0x7fdc9c0c5de0 2024-03-27T03:13:48.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.378+0000 7fdcaacd6640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:48.371 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.378+0000 7fdcaacd6640 10 monclient(hunting): authenticate will time out at 1788.683823s 2024-03-27T03:13:48.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.382+0000 7fdcaa4d5640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdc9c0c3070 0x7fdc9c0d0fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:38471/0 (socket says 172.21.15.153:38471) 2024-03-27T03:13:48.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.382+0000 7fdcaa4d5640 1 -- 172.21.15.153:0/2696484788 learned_addr learned my addr 172.21.15.153:0/2696484788 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:48.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285975821 0 0) 0x7fdc9c02e040 con 0x7fdc9c0c3070 2024-03-27T03:13:48.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 10 monclient(hunting): _negotiate 2024-03-27T03:13:48.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:48.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:48.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 10 monclient(hunting): my global_id is 4525 2024-03-27T03:13:48.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc90003520 con 0x7fdc9c0c3070 2024-03-27T03:13:48.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587393499 0 0) 0x7fdc9c02e320 con 0x7fdc9c0c5de0 2024-03-27T03:13:48.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 10 monclient(hunting): _negotiate 2024-03-27T03:13:48.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:48.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:48.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 10 monclient(hunting): my global_id is 4568 2024-03-27T03:13:48.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc9c02e040 con 0x7fdc9c0c5de0 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576181055 0 0) 0x7fdc9c02cc60 con 0x7fdc9c036a60 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 10 monclient(hunting): _negotiate 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 10 monclient(hunting): my global_id is 4497 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.383+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc9c02e320 con 0x7fdc9c036a60 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 96973900 0 0) 0x7fdc9c02e320 con 0x7fdc9c036a60 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 >> v1:172.21.15.153:6790/0 conn(0x7fdc9c0c5de0 legacy=0x7fdc9c0c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 >> v1:172.21.15.196:6789/0 conn(0x7fdc9c0c3070 legacy=0x7fdc9c0d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 10 monclient: _finish_hunting 0 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 1 monclient: found mon.noname-a 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c02c830 con 0x7fdc9c036a60 2024-03-27T03:13:48.381 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 10 monclient: _finish_auth 0 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 10 monclient: _check_auth_tickets 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdcaacd6640 5 monclient: authenticate success, global_id 4497 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdcaacd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdc94001630 con 0x7fdc9c036a60 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 10 monclient: dump: 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:48.382 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdcaacd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc94001880 con 0x7fdc9c036a60 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 1 -- 172.21.15.153:0/2696484788 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdc94001c10 con 0x7fdc9c036a60 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdca8cd2640 10 monclient: dump: 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:48.383 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:48.384 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:48.384 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:48.384 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:48.384 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:48.384 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:48.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdcaacd6640 10 monclient: get_monmap_and_config success 2024-03-27T03:13:48.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.384+0000 7fdcaacd6640 4 set_mon_vals no callback set 2024-03-27T03:13:48.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.385+0000 7fdcaacd6640 10 monclient: shutdown 2024-03-27T03:13:48.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.385+0000 7fdcaacd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:48.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.385+0000 7fdcaacd6640 1 -- 172.21.15.153:0/2696484788 >> v1:172.21.15.153:6789/0 conn(0x7fdc9c036a60 legacy=0x7fdc9c0c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:48.386 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.388+0000 7fdcaacd6640 1 -- 172.21.15.153:0/2696484788 shutdown_connections 2024-03-27T03:13:48.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.388+0000 7fdcaacd6640 1 -- 172.21.15.153:0/2696484788 >> 172.21.15.153:0/2696484788 conn(0x7fdc9c005c00 msgr2=0x7fdc9c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:48.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.389+0000 7fdcaacd6640 1 -- 172.21.15.153:0/2696484788 shutdown_connections 2024-03-27T03:13:48.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.389+0000 7fdcaacd6640 1 -- 172.21.15.153:0/2696484788 wait complete. 2024-03-27T03:13:48.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.390+0000 7fdcaacd6640 10 monclient: build_initial_monmap 2024-03-27T03:13:48.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.390+0000 7fdcaacd6640 10 monclient: monmap: 2024-03-27T03:13:48.387 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:48.387 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:13:48.387 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:13:48.387 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:13:48.387 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:48.387 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:48.387 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.390+0000 7fdcaacd6640 1 librados: starting msgr at 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.390+0000 7fdcaacd6640 1 librados: starting objecter 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.390+0000 7fdcaacd6640 1 Processor -- start 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.390+0000 7fdcaacd6640 1 -- start start 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.390+0000 7fdcaacd6640 1 librados: setting wanted keys 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.390+0000 7fdcaacd6640 1 librados: calling monclient init 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.390+0000 7fdcaacd6640 10 monclient: init 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.391+0000 7fdcaacd6640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.391+0000 7fdcaacd6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.391+0000 7fdcaacd6640 10 monclient(hunting): picked mon.noname-b con 0x7fdc9c036a60 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.391+0000 7fdcaacd6640 10 monclient(hunting): picked mon.noname-a con 0x7fdc9c0c3070 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.391+0000 7fdcaacd6640 10 monclient(hunting): picked mon.noname-c con 0x7fdc9c15ed30 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:48.388 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.391+0000 7fdcaacd6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c0c4c30 con 0x7fdc9c0c3070 2024-03-27T03:13:48.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.391+0000 7fdcaacd6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c0c4ea0 con 0x7fdc9c036a60 2024-03-27T03:13:48.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.391+0000 7fdcaacd6640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c008980 con 0x7fdc9c15ed30 2024-03-27T03:13:48.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.391+0000 7fdcaacd6640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:48.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.391+0000 7fdcaacd6640 10 monclient(hunting): authenticate will time out at 1788.697169s 2024-03-27T03:13:48.390 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.397+0000 7fdca9cd4640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdc9c036a60 0x7fdc9c169570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:38539/0 (socket says 172.21.15.153:38539) 2024-03-27T03:13:48.390 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.397+0000 7fdca9cd4640 1 -- 172.21.15.153:0/1592377617 learned_addr learned my addr 172.21.15.153:0/1592377617 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:48.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.398+0000 7fdc9a7fc640 10 client.?.objecter ms_handle_connect 0x7fdc9c036a60 2024-03-27T03:13:48.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.398+0000 7fdc9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:48.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.398+0000 7fdc9a7fc640 10 client.?.objecter ms_handle_connect 0x7fdc9c15ed30 2024-03-27T03:13:48.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.398+0000 7fdc9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:48.392 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.399+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949640909 0 0) 0x7fdc9c0c4ea0 con 0x7fdc9c036a60 2024-03-27T03:13:48.393 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.399+0000 7fdc9a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:13:48.393 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.399+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:48.393 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.399+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:48.393 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.399+0000 7fdc9a7fc640 10 monclient(hunting): my global_id is 4528 2024-03-27T03:13:48.393 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.399+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc88004eb0 con 0x7fdc9c036a60 2024-03-27T03:13:48.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2690081283 0 0) 0x7fdc88004eb0 con 0x7fdc9c036a60 2024-03-27T03:13:48.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc9c0c4ea0 con 0x7fdc9c036a60 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdc94001c90 con 0x7fdc9c036a60 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 10 monclient(hunting): dump: 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:48.397 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508635355 0 0) 0x7fdc9c008980 con 0x7fdc9c15ed30 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 10 monclient(hunting): my global_id is 4571 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.400+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc88004eb0 con 0x7fdc9c15ed30 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2203181023 0 0) 0x7fdc9c0c4ea0 con 0x7fdc9c036a60 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 >> v1:172.21.15.153:6790/0 conn(0x7fdc9c15ed30 legacy=0x7fdc9c173fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 >> v1:172.21.15.153:6789/0 conn(0x7fdc9c0c3070 legacy=0x7fdc9c0c3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdc9a7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdc9a7fc640 1 monclient: found mon.b 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdc9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:48.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdc9a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c0c52c0 con 0x7fdc9c036a60 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdc9a7fc640 10 monclient: _finish_auth 0 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdc9a7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdc9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdcaacd6640 5 monclient: authenticate success, global_id 4528 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdcaacd6640 10 monclient: _renew_subs 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdcaacd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdcaacd6640 1 -- 172.21.15.153:0/1592377617 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9c0c5510 con 0x7fdc9c036a60 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdcaacd6640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdcaacd6640 10 monclient: _renew_subs 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdcaacd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdcaacd6640 1 -- 172.21.15.153:0/1592377617 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9c1758b0 con 0x7fdc9c036a60 2024-03-27T03:13:48.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.401+0000 7fdcaacd6640 1 librados: init done 2024-03-27T03:13:48.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.402+0000 7fdcaacd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:48.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.403+0000 7fdcaacd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:48.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.403+0000 7fdcaacd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:48.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.403+0000 7fdcaacd6640 1 -- 172.21.15.153:0/1592377617 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc9c005120 con 0x7fdc9c036a60 2024-03-27T03:13:48.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.405+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc94003a60 con 0x7fdc9c036a60 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.405+0000 7fdc9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.405+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdc94003df0 con 0x7fdc9c036a60 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.405+0000 7fdc9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.405+0000 7fdc9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.405+0000 7fdc9a7fc640 10 monclient: dump: 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:48.402 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:48.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.406+0000 7fdc9b7fe640 4 set_mon_vals no callback set 2024-03-27T03:13:48.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.407+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fdc94003f50 con 0x7fdc9c036a60 2024-03-27T03:13:48.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.408+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fdc94084cd0 con 0x7fdc9c036a60 2024-03-27T03:13:48.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.408+0000 7fdc9a7fc640 10 client.4528.objecter ms_dispatch 0x7fdca4000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:13:48.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.408+0000 7fdc9a7fc640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:13:48.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.408+0000 7fdc9a7fc640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:13:48.403 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.408+0000 7fdc9a7fc640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-27T03:13:48.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.409+0000 7fdca94d3640 10 monclient: get_auth_request con 0x7fdc8806dbd0 auth_method 0 2024-03-27T03:13:48.404 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.410+0000 7fdc9a7fc640 10 client.4528.objecter ms_handle_connect 0x7fdc8806dbd0 2024-03-27T03:13:48.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.414+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc940505b0 con 0x7fdc9c036a60 2024-03-27T03:13:48.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.414+0000 7fdc9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:48.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.414+0000 7fdc9a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:13:48.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.734+0000 7fdcaacd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:48.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.734+0000 7fdcaacd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:48.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.734+0000 7fdcaacd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:48.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.734+0000 7fdcaacd6640 1 -- 172.21.15.153:0/1592377617 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc9c0c34d0 con 0x7fdc9c036a60 2024-03-27T03:13:48.728 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.735+0000 7fdc9a7fc640 1 -- 172.21.15.153:0/1592377617 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fdc940524e0 con 0x7fdc9c036a60 2024-03-27T03:13:48.728 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.735+0000 7fdc9a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:48.728 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.735+0000 7fdc9a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:13:48.728 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:13:48.728 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:13:48.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.738+0000 7fdcaacd6640 10 librados: watch_flush enter 2024-03-27T03:13:48.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.738+0000 7fdcaacd6640 10 librados: watch_flush exit 2024-03-27T03:13:48.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.738+0000 7fdcaacd6640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-27T03:13:48.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.738+0000 7fdcaacd6640 10 client.4528.objecter successfully canceled tick 2024-03-27T03:13:48.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.739+0000 7fdcaacd6640 1 -- 172.21.15.153:0/1592377617 >> v1:172.21.15.153:6816/2637937781 conn(0x7fdc8806dbd0 legacy=0x7fdc88070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:48.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.739+0000 7fdcaacd6640 10 monclient: shutdown 2024-03-27T03:13:48.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.739+0000 7fdcaacd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:48.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.739+0000 7fdcaacd6640 1 -- 172.21.15.153:0/1592377617 >> v1:172.21.15.196:6789/0 conn(0x7fdc9c036a60 legacy=0x7fdc9c169570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:48.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.740+0000 7fdcaacd6640 1 -- 172.21.15.153:0/1592377617 shutdown_connections 2024-03-27T03:13:48.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.740+0000 7fdcaacd6640 1 -- 172.21.15.153:0/1592377617 >> 172.21.15.153:0/1592377617 conn(0x7fdc9c005c00 msgr2=0x7fdc9c0c2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:48.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.742+0000 7fdcaacd6640 1 -- 172.21.15.153:0/1592377617 shutdown_connections 2024-03-27T03:13:48.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.742+0000 7fdcaacd6640 1 -- 172.21.15.153:0/1592377617 wait complete. 2024-03-27T03:13:48.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:48.742+0000 7fdcaacd6640 1 librados: shutdown 2024-03-27T03:13:48.756 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:13:55.563 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:13:55.717 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.724+0000 7f9677554640 10 monclient: get_monmap_and_config 2024-03-27T03:13:55.717 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.724+0000 7f9677554640 10 monclient: build_initial_monmap 2024-03-27T03:13:55.717 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.724+0000 7f9677554640 10 monclient: monmap: 2024-03-27T03:13:55.718 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:55.718 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:55.718 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:13:55.725548+0000 2024-03-27T03:13:55.718 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:13:55.725548+0000 2024-03-27T03:13:55.718 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:55.718 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:55.718 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:55.718 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:13:55.718 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:13:55.718 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:55.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.725+0000 7f9677554640 1 Processor -- start 2024-03-27T03:13:55.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.725+0000 7f9677554640 1 -- start start 2024-03-27T03:13:55.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.725+0000 7f9677554640 10 monclient: init 2024-03-27T03:13:55.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.726+0000 7f9677554640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:55.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.726+0000 7f9677554640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:13:55.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.726+0000 7f9677554640 10 monclient(hunting): picked mon.noname-c con 0x7f96700b48a0 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:55.724 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.727+0000 7f9677554640 10 monclient(hunting): picked mon.noname-a con 0x7f96700b3f70 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:55.724 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.728+0000 7f9677554640 10 monclient(hunting): picked mon.noname-b con 0x7f967013fb20 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:55.730 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.728+0000 7f9677554640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96700b43b0 con 0x7f96700b3f70 2024-03-27T03:13:55.730 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.728+0000 7f9677554640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96700b0b90 con 0x7f96700b48a0 2024-03-27T03:13:55.730 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.728+0000 7f9677554640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96700b19a0 con 0x7f967013fb20 2024-03-27T03:13:55.730 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.728+0000 7f9677554640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:55.730 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.728+0000 7f9677554640 10 monclient(hunting): authenticate will time out at 1796.033931s 2024-03-27T03:13:55.730 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.728+0000 7f9675af1640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f967013fb20 0x7f967014da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:42392/0 (socket says 172.21.15.153:42392) 2024-03-27T03:13:55.730 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.728+0000 7f9675af1640 1 -- 172.21.15.153:0/3365565914 learned_addr learned my addr 172.21.15.153:0/3365565914 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:55.730 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 1 -- 172.21.15.153:0/3365565914 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407871430 0 0) 0x7f96700b19a0 con 0x7f967013fb20 2024-03-27T03:13:55.730 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient(hunting): _negotiate 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient(hunting): my global_id is 4574 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 1 -- 172.21.15.153:0/3365565914 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f965c003520 con 0x7f967013fb20 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 1 -- 172.21.15.153:0/3365565914 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3190405857 0 0) 0x7f965c003520 con 0x7f967013fb20 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 1 -- 172.21.15.153:0/3365565914 >> v1:172.21.15.196:6789/0 conn(0x7f96700b48a0 legacy=0x7f96700b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 1 -- 172.21.15.153:0/3365565914 >> v1:172.21.15.153:6789/0 conn(0x7f96700b3f70 legacy=0x7f967013f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient: _finish_hunting 0 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 1 monclient: found mon.noname-b 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 1 -- 172.21.15.153:0/3365565914 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96700af800 con 0x7f967013fb20 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient: _finish_auth 0 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient: _check_auth_tickets 2024-03-27T03:13:55.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 1 -- 172.21.15.153:0/3365565914 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9660001690 con 0x7f967013fb20 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient: dump: 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:55.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9677554640 5 monclient: authenticate success, global_id 4574 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9677554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 1 -- 172.21.15.153:0/3365565914 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9660001860 con 0x7f967013fb20 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 1 -- 172.21.15.153:0/3365565914 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9660001cb0 con 0x7f967013fb20 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9667fff640 10 monclient: dump: 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:55.733 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9677554640 10 monclient: get_monmap_and_config success 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.730+0000 7f9677554640 4 set_mon_vals no callback set 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.731+0000 7f9677554640 10 monclient: shutdown 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.731+0000 7f9677554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.731+0000 7f9677554640 1 -- 172.21.15.153:0/3365565914 >> v1:172.21.15.153:6790/0 conn(0x7f967013fb20 legacy=0x7f967014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.732+0000 7f9677554640 1 -- 172.21.15.153:0/3365565914 shutdown_connections 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.732+0000 7f9677554640 1 -- 172.21.15.153:0/3365565914 >> 172.21.15.153:0/3365565914 conn(0x7f967006c8a0 msgr2=0x7f967006cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.733+0000 7f9677554640 1 -- 172.21.15.153:0/3365565914 shutdown_connections 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.733+0000 7f9677554640 1 -- 172.21.15.153:0/3365565914 wait complete. 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.735+0000 7f9677554640 10 monclient: build_initial_monmap 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.735+0000 7f9677554640 10 monclient: monmap: 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:13:55.734 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.735+0000 7f9677554640 1 librados: starting msgr at 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.735+0000 7f9677554640 1 librados: starting objecter 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.735+0000 7f9677554640 1 Processor -- start 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.735+0000 7f9677554640 1 -- start start 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.735+0000 7f9677554640 1 librados: setting wanted keys 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.735+0000 7f9677554640 1 librados: calling monclient init 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.735+0000 7f9677554640 10 monclient: init 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.736+0000 7f9677554640 10 monclient: _reopen_session rank -1 2024-03-27T03:13:55.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.736+0000 7f9677554640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:13:55.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.736+0000 7f9677554640 10 monclient(hunting): picked mon.noname-c con 0x7f96700b48a0 addr v1:172.21.15.153:6790/0 2024-03-27T03:13:55.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.736+0000 7f9677554640 10 monclient(hunting): picked mon.noname-a con 0x7f967013fb20 addr v1:172.21.15.153:6789/0 2024-03-27T03:13:55.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.736+0000 7f9677554640 10 monclient(hunting): picked mon.noname-b con 0x7f9670141b10 addr v1:172.21.15.196:6789/0 2024-03-27T03:13:55.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.736+0000 7f9677554640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9670142b20 con 0x7f967013fb20 2024-03-27T03:13:55.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.736+0000 7f9677554640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96701dff70 con 0x7f9670141b10 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.736+0000 7f9677554640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9670145f10 con 0x7f96700b48a0 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.736+0000 7f9677554640 10 monclient(hunting): _renew_subs 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.736+0000 7f9677554640 10 monclient(hunting): authenticate will time out at 1796.042514s 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.742+0000 7f9675af1640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f9670141b10 0x7f96701e7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:58901/0 (socket says 172.21.15.153:58901) 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.742+0000 7f9675af1640 1 -- 172.21.15.153:0/2293951488 learned_addr learned my addr 172.21.15.153:0/2293951488 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.742+0000 7f9665ffb640 10 client.?.objecter ms_handle_connect 0x7f9670141b10 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.742+0000 7f9665ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392793935 0 0) 0x7f96701dff70 con 0x7f9670141b10 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 10 monclient(hunting): my global_id is 4531 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9654004eb0 con 0x7f9670141b10 2024-03-27T03:13:55.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1231879074 0 0) 0x7f9654004eb0 con 0x7f9670141b10 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96701dff70 con 0x7f9670141b10 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9660001bb0 con 0x7f9670141b10 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 10 monclient(hunting): dump: 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:55.739 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:55.741 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:13:55.741 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1291537639 0 0) 0x7f96701dff70 con 0x7f9670141b10 2024-03-27T03:13:55.741 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 >> v1:172.21.15.153:6790/0 conn(0x7f96700b48a0 legacy=0x7f9670141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:55.741 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.743+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 >> v1:172.21.15.153:6789/0 conn(0x7f967013fb20 legacy=0x7f9670070710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:13:55.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9665ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:13:55.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9665ffb640 1 monclient: found mon.b 2024-03-27T03:13:55.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9665ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:13:55.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9665ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:55.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96701461f0 con 0x7f9670141b10 2024-03-27T03:13:55.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9665ffb640 10 monclient: _finish_auth 0 2024-03-27T03:13:55.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9665ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:13:55.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9665ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:13:55.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9677554640 5 monclient: authenticate success, global_id 4531 2024-03-27T03:13:55.742 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9677554640 10 monclient: _renew_subs 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9677554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9677554640 1 -- 172.21.15.153:0/2293951488 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96701dfa50 con 0x7f9670141b10 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9677554640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9677554640 10 monclient: _renew_subs 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9677554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9677554640 1 -- 172.21.15.153:0/2293951488 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9670144330 con 0x7f9670141b10 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9677554640 1 librados: init done 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9660001d10 con 0x7f9670141b10 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9665ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f9660004c30 con 0x7f9670141b10 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.746+0000 7f9665ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.747+0000 7f9665ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.747+0000 7f9665ffb640 10 monclient: dump: 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:13:55.743 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:13:55.744 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:55.744 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:13:55.744 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:13:55.744 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:13:55.744 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:13:55.744 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:13:55.744 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:13:55.744 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:13:55.745 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.748+0000 7f9666ffd640 4 set_mon_vals no callback set 2024-03-27T03:13:55.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.750+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f9660005e80 con 0x7f9670141b10 2024-03-27T03:13:55.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.751+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f9660084bc0 con 0x7f9670141b10 2024-03-27T03:13:55.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.751+0000 7f9665ffb640 10 client.4531.objecter ms_dispatch 0x7f9670000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:13:55.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.752+0000 7f9665ffb640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:13:55.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.752+0000 7f9665ffb640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:13:55.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.752+0000 7f9665ffb640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-27T03:13:55.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.752+0000 7f9677554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:55.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.752+0000 7f96752f0640 10 monclient: get_auth_request con 0x7f965406dc70 auth_method 0 2024-03-27T03:13:55.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.752+0000 7f9677554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:55.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.752+0000 7f9677554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:55.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.752+0000 7f9677554640 1 -- 172.21.15.153:0/2293951488 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9650005080 con 0x7f9670141b10 2024-03-27T03:13:55.746 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.753+0000 7f9665ffb640 10 client.4531.objecter ms_handle_connect 0x7f965406dc70 2024-03-27T03:13:55.755 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.762+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96600502d0 con 0x7f9670141b10 2024-03-27T03:13:55.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.762+0000 7f9665ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:13:55.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:55.762+0000 7f9665ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:13:56.118 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.125+0000 7f9677554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:56.118 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.125+0000 7f9677554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:56.118 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.125+0000 7f9677554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:13:56.119 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.125+0000 7f9677554640 1 -- 172.21.15.153:0/2293951488 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9650005330 con 0x7f9670141b10 2024-03-27T03:13:56.119 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.126+0000 7f9665ffb640 1 -- 172.21.15.153:0/2293951488 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f96600523d0 con 0x7f9670141b10 2024-03-27T03:13:56.119 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.126+0000 7f9665ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:13:56.119 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.126+0000 7f9665ffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:13:56.119 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:13:56.120 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:13:56.123 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.130+0000 7f9677554640 10 librados: watch_flush enter 2024-03-27T03:13:56.123 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.130+0000 7f9677554640 10 librados: watch_flush exit 2024-03-27T03:13:56.123 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.130+0000 7f9677554640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-27T03:13:56.123 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.130+0000 7f9677554640 10 client.4531.objecter successfully canceled tick 2024-03-27T03:13:56.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.130+0000 7f9677554640 1 -- 172.21.15.153:0/2293951488 >> v1:172.21.15.153:6816/2637937781 conn(0x7f965406dc70 legacy=0x7f9654070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:56.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.130+0000 7f9677554640 10 monclient: shutdown 2024-03-27T03:13:56.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.130+0000 7f9677554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:13:56.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.130+0000 7f9677554640 1 -- 172.21.15.153:0/2293951488 >> v1:172.21.15.196:6789/0 conn(0x7f9670141b10 legacy=0x7f96701e7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:13:56.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.131+0000 7f9677554640 1 -- 172.21.15.153:0/2293951488 shutdown_connections 2024-03-27T03:13:56.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.131+0000 7f9677554640 1 -- 172.21.15.153:0/2293951488 >> 172.21.15.153:0/2293951488 conn(0x7f967006c8a0 msgr2=0x7f967013c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:13:56.124 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.131+0000 7f9677554640 1 -- 172.21.15.153:0/2293951488 shutdown_connections 2024-03-27T03:13:56.125 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.131+0000 7f9677554640 1 -- 172.21.15.153:0/2293951488 wait complete. 2024-03-27T03:13:56.125 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:13:56.131+0000 7f9677554640 1 librados: shutdown 2024-03-27T03:13:56.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:14:02.946 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:14:03.098 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.105+0000 7fec592c0640 10 monclient: get_monmap_and_config 2024-03-27T03:14:03.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.105+0000 7fec592c0640 10 monclient: build_initial_monmap 2024-03-27T03:14:03.099 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.105+0000 7fec592c0640 10 monclient: monmap: 2024-03-27T03:14:03.099 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:03.099 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:03.099 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:14:03.106373+0000 2024-03-27T03:14:03.099 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:14:03.106373+0000 2024-03-27T03:14:03.099 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:03.099 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:03.099 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:03.099 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:14:03.099 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:14:03.099 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:03.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.107+0000 7fec592c0640 1 Processor -- start 2024-03-27T03:14:03.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.107+0000 7fec592c0640 1 -- start start 2024-03-27T03:14:03.100 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.107+0000 7fec592c0640 10 monclient: init 2024-03-27T03:14:03.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.108+0000 7fec592c0640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:03.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.108+0000 7fec592c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:14:03.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.108+0000 7fec592c0640 10 monclient(hunting): picked mon.noname-c con 0x7fec540a9ac0 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:03.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.108+0000 7fec592c0640 10 monclient(hunting): picked mon.noname-a con 0x7fec5413ee90 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:03.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.108+0000 7fec592c0640 10 monclient(hunting): picked mon.noname-b con 0x7fec540af900 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:03.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.111+0000 7fec592c0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec540afd10 con 0x7fec5413ee90 2024-03-27T03:14:03.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.111+0000 7fec592c0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec540af6f0 con 0x7fec540a9ac0 2024-03-27T03:14:03.104 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.111+0000 7fec592c0640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec540aac80 con 0x7fec540af900 2024-03-27T03:14:03.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.111+0000 7fec592c0640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:03.105 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.111+0000 7fec592c0640 10 monclient(hunting): authenticate will time out at 1803.416962s 2024-03-27T03:14:03.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.114+0000 7fec537fe640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fec540af900 0x7fec54149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:60117/0 (socket says 172.21.15.153:60117) 2024-03-27T03:14:03.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.114+0000 7fec537fe640 1 -- 172.21.15.153:0/2189727797 learned_addr learned my addr 172.21.15.153:0/2189727797 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:03.107 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712492429 0 0) 0x7fec540aac80 con 0x7fec540af900 2024-03-27T03:14:03.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:14:03.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:03.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:03.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 10 monclient(hunting): my global_id is 4577 2024-03-27T03:14:03.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec3c003520 con 0x7fec540af900 2024-03-27T03:14:03.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232441236 0 0) 0x7fec540af6f0 con 0x7fec540a9ac0 2024-03-27T03:14:03.108 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:14:03.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:03.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:03.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 10 monclient(hunting): my global_id is 4534 2024-03-27T03:14:03.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec540aac80 con 0x7fec540a9ac0 2024-03-27T03:14:03.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257391358 0 0) 0x7fec540afd10 con 0x7fec5413ee90 2024-03-27T03:14:03.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:14:03.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:03.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:03.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 10 monclient(hunting): my global_id is 4500 2024-03-27T03:14:03.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.115+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec540af6f0 con 0x7fec5413ee90 2024-03-27T03:14:03.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1009320783 0 0) 0x7fec540aac80 con 0x7fec540a9ac0 2024-03-27T03:14:03.113 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 >> v1:172.21.15.153:6790/0 conn(0x7fec540af900 legacy=0x7fec54149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 >> v1:172.21.15.153:6789/0 conn(0x7fec5413ee90 legacy=0x7fec5413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 1 monclient: found mon.noname-c 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec540aaf60 con 0x7fec540a9ac0 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 10 monclient: _finish_auth 0 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fec4c0018a0 con 0x7fec540a9ac0 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:14:03.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 10 monclient: dump: 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec4c001ac0 con 0x7fec540a9ac0 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 1 -- 172.21.15.153:0/2189727797 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fec4c001e50 con 0x7fec540a9ac0 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:03.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:14:03.116 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.119+0000 7fec51ffb640 10 monclient: dump: 2024-03-27T03:14:03.116 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:03.116 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:03.116 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:03.116 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:03.116 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:03.116 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:03.116 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:03.116 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:03.116 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:03.116 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:03.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.121+0000 7fec592c0640 5 monclient: authenticate success, global_id 4534 2024-03-27T03:14:03.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.121+0000 7fec592c0640 10 monclient: get_monmap_and_config success 2024-03-27T03:14:03.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.121+0000 7fec592c0640 4 set_mon_vals no callback set 2024-03-27T03:14:03.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.122+0000 7fec592c0640 10 monclient: shutdown 2024-03-27T03:14:03.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.122+0000 7fec592c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:03.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.122+0000 7fec592c0640 1 -- 172.21.15.153:0/2189727797 >> v1:172.21.15.196:6789/0 conn(0x7fec540a9ac0 legacy=0x7fec540adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:03.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.123+0000 7fec592c0640 1 -- 172.21.15.153:0/2189727797 shutdown_connections 2024-03-27T03:14:03.117 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.123+0000 7fec592c0640 1 -- 172.21.15.153:0/2189727797 >> 172.21.15.153:0/2189727797 conn(0x7fec540b35e0 msgr2=0x7fec540b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:03.118 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.125+0000 7fec592c0640 1 -- 172.21.15.153:0/2189727797 shutdown_connections 2024-03-27T03:14:03.119 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.125+0000 7fec592c0640 1 -- 172.21.15.153:0/2189727797 wait complete. 2024-03-27T03:14:03.119 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.126+0000 7fec592c0640 10 monclient: build_initial_monmap 2024-03-27T03:14:03.119 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.126+0000 7fec592c0640 10 monclient: monmap: 2024-03-27T03:14:03.119 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:03.119 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:14:03.119 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:14:03.119 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:14:03.119 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:03.119 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:03.119 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:03.120 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:14:03.120 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:14:03.120 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:03.121 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.127+0000 7fec592c0640 1 librados: starting msgr at 2024-03-27T03:14:03.121 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.127+0000 7fec592c0640 1 librados: starting objecter 2024-03-27T03:14:03.121 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.127+0000 7fec592c0640 1 Processor -- start 2024-03-27T03:14:03.121 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.127+0000 7fec592c0640 1 -- start start 2024-03-27T03:14:03.121 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.128+0000 7fec592c0640 1 librados: setting wanted keys 2024-03-27T03:14:03.121 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.128+0000 7fec592c0640 1 librados: calling monclient init 2024-03-27T03:14:03.121 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.128+0000 7fec592c0640 10 monclient: init 2024-03-27T03:14:03.122 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.129+0000 7fec592c0640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:03.122 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.129+0000 7fec592c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:14:03.125 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.132+0000 7fec52ffd640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fec540a9ac0 0x7fec54139860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:60239/0 (socket says 172.21.15.153:60239) 2024-03-27T03:14:03.125 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.132+0000 7fec52ffd640 1 -- 172.21.15.153:0/2802013788 learned_addr learned my addr 172.21.15.153:0/2802013788 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:03.125 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.132+0000 7fec592c0640 10 monclient(hunting): picked mon.noname-c con 0x7fec540a9ac0 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:03.126 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.134+0000 7fec3b7fe640 10 client.?.objecter ms_handle_connect 0x7fec540a9ac0 2024-03-27T03:14:03.127 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.134+0000 7fec3b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:03.128 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.135+0000 7fec592c0640 10 monclient(hunting): picked mon.noname-b con 0x7fec540af900 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:03.128 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.135+0000 7fec592c0640 10 monclient(hunting): picked mon.noname-a con 0x7fec5413ee90 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:03.128 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.135+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5406f490 con 0x7fec5413ee90 2024-03-27T03:14:03.128 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.135+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5406f000 con 0x7fec540af900 2024-03-27T03:14:03.128 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.135+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec54070710 con 0x7fec540a9ac0 2024-03-27T03:14:03.129 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.135+0000 7fec592c0640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:03.129 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.135+0000 7fec592c0640 10 monclient(hunting): authenticate will time out at 1803.440893s 2024-03-27T03:14:03.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 10 client.?.objecter ms_handle_connect 0x7fec540af900 2024-03-27T03:14:03.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:03.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460341542 0 0) 0x7fec54070710 con 0x7fec540a9ac0 2024-03-27T03:14:03.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:14:03.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:03.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:03.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 10 monclient(hunting): my global_id is 4580 2024-03-27T03:14:03.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec40004eb0 con 0x7fec540a9ac0 2024-03-27T03:14:03.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143654962 0 0) 0x7fec5406f000 con 0x7fec540af900 2024-03-27T03:14:03.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 10 monclient(hunting): _negotiate 2024-03-27T03:14:03.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:03.136 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.138+0000 7fec3b7fe640 10 monclient(hunting): my global_id is 4537 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.139+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec54070710 con 0x7fec540af900 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.139+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1609500063 0 0) 0x7fec40004eb0 con 0x7fec540a9ac0 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.139+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec40005490 con 0x7fec540a9ac0 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.139+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fec4c0018a0 con 0x7fec540a9ac0 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.139+0000 7fec3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.139+0000 7fec3b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.139+0000 7fec3b7fe640 10 monclient(hunting): dump: 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:03.137 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.139+0000 7fec3b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.139+0000 7fec3b7fe640 10 client.?.objecter ms_handle_connect 0x7fec5413ee90 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.139+0000 7fec3b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.139+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 149455043 0 0) 0x7fec40005490 con 0x7fec540a9ac0 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 >> v1:172.21.15.196:6789/0 conn(0x7fec540af900 legacy=0x7fec541d6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 >> v1:172.21.15.153:6789/0 conn(0x7fec5413ee90 legacy=0x7fec541d6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 10 monclient: _finish_hunting 0 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 1 monclient: found mon.c 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5406f640 con 0x7fec540a9ac0 2024-03-27T03:14:03.138 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 10 monclient: _finish_auth 0 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 10 monclient: _check_auth_tickets 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec4c003c20 con 0x7fec540a9ac0 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fec4c003ff0 con 0x7fec540a9ac0 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.140+0000 7fec3b7fe640 10 monclient: dump: 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:03.139 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.141+0000 7fec517fa640 4 set_mon_vals no callback set 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.141+0000 7fec592c0640 5 monclient: authenticate success, global_id 4580 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.141+0000 7fec592c0640 10 monclient: _renew_subs 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.141+0000 7fec592c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.141+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec541423e0 con 0x7fec540a9ac0 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.141+0000 7fec592c0640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.141+0000 7fec592c0640 10 monclient: _renew_subs 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.141+0000 7fec592c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.141+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec54071060 con 0x7fec540a9ac0 2024-03-27T03:14:03.140 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.141+0000 7fec592c0640 1 librados: init done 2024-03-27T03:14:03.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.145+0000 7fec592c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:03.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.145+0000 7fec592c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:03.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.145+0000 7fec592c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:03.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.145+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5413bcd0 con 0x7fec540a9ac0 2024-03-27T03:14:03.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.146+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fec4c004150 con 0x7fec540a9ac0 2024-03-27T03:14:03.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.147+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fec4c084ee0 con 0x7fec540a9ac0 2024-03-27T03:14:03.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.147+0000 7fec3b7fe640 10 client.4580.objecter ms_dispatch 0x7fec54000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:14:03.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.147+0000 7fec3b7fe640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:14:03.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.147+0000 7fec3b7fe640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:14:03.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.147+0000 7fec3b7fe640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-27T03:14:03.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.147+0000 7fec527fc640 10 monclient: get_auth_request con 0x7fec4006de80 auth_method 0 2024-03-27T03:14:03.142 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.147+0000 7fec3b7fe640 10 client.4580.objecter ms_handle_connect 0x7fec4006de80 2024-03-27T03:14:03.150 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.157+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec4c0505f0 con 0x7fec540a9ac0 2024-03-27T03:14:03.150 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.157+0000 7fec3b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:03.150 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.157+0000 7fec3b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:14:03.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.477+0000 7fec592c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:03.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.477+0000 7fec592c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:03.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.477+0000 7fec592c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:03.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.477+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec54075f10 con 0x7fec540a9ac0 2024-03-27T03:14:03.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.481+0000 7fec3b7fe640 1 -- 172.21.15.153:0/2802013788 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fec4c0526f0 con 0x7fec540a9ac0 2024-03-27T03:14:03.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.481+0000 7fec3b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:03.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.481+0000 7fec3b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:14:03.475 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:14:03.475 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:14:03.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.483+0000 7fec592c0640 10 librados: watch_flush enter 2024-03-27T03:14:03.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.483+0000 7fec592c0640 10 librados: watch_flush exit 2024-03-27T03:14:03.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.483+0000 7fec592c0640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-27T03:14:03.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.483+0000 7fec592c0640 10 client.4580.objecter successfully canceled tick 2024-03-27T03:14:03.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.484+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 >> v1:172.21.15.153:6816/2637937781 conn(0x7fec4006de80 legacy=0x7fec40070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:03.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.484+0000 7fec592c0640 10 monclient: shutdown 2024-03-27T03:14:03.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.484+0000 7fec592c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:03.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.484+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 >> v1:172.21.15.153:6790/0 conn(0x7fec540a9ac0 legacy=0x7fec54139860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:03.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.484+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 shutdown_connections 2024-03-27T03:14:03.478 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.484+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 >> 172.21.15.153:0/2802013788 conn(0x7fec540b35e0 msgr2=0x7fec5413b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:03.478 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.485+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 shutdown_connections 2024-03-27T03:14:03.480 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.486+0000 7fec592c0640 1 -- 172.21.15.153:0/2802013788 wait complete. 2024-03-27T03:14:03.480 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:03.487+0000 7fec592c0640 1 librados: shutdown 2024-03-27T03:14:03.501 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:14:10.308 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:14:10.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.471+0000 7f5c910fb640 10 monclient: get_monmap_and_config 2024-03-27T03:14:10.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.471+0000 7f5c910fb640 10 monclient: build_initial_monmap 2024-03-27T03:14:10.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.471+0000 7f5c910fb640 10 monclient: monmap: 2024-03-27T03:14:10.465 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:10.465 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:10.465 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:14:10.472491+0000 2024-03-27T03:14:10.465 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:14:10.472491+0000 2024-03-27T03:14:10.465 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:10.465 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:10.465 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:10.465 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:14:10.466 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:14:10.466 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:10.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.473+0000 7f5c910fb640 1 Processor -- start 2024-03-27T03:14:10.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.473+0000 7f5c910fb640 1 -- start start 2024-03-27T03:14:10.466 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.473+0000 7f5c910fb640 10 monclient: init 2024-03-27T03:14:10.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.474+0000 7f5c910fb640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:10.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.474+0000 7f5c910fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:14:10.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.474+0000 7f5c910fb640 10 monclient(hunting): picked mon.noname-a con 0x7f5c8c0b1f60 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:10.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.477+0000 7f5c910fb640 10 monclient(hunting): picked mon.noname-c con 0x7f5c8c071eb0 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:10.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c910fb640 10 monclient(hunting): picked mon.noname-b con 0x7f5c8c143d10 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:10.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c910fb640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c06ef80 con 0x7f5c8c0b1f60 2024-03-27T03:14:10.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c910fb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c06f260 con 0x7f5c8c071eb0 2024-03-27T03:14:10.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c910fb640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c0b40c0 con 0x7f5c8c143d10 2024-03-27T03:14:10.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c910fb640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:10.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c910fb640 10 monclient(hunting): authenticate will time out at 1810.783628s 2024-03-27T03:14:10.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c8a59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f5c8c071eb0 0x7f5c8c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:44125/0 (socket says 172.21.15.153:44125) 2024-03-27T03:14:10.471 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c8a59c640 1 -- 172.21.15.153:0/386069998 learned_addr learned my addr 172.21.15.153:0/386069998 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:10.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c89d9b640 1 -- 172.21.15.153:0/386069998 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091826980 0 0) 0x7f5c8c06f260 con 0x7f5c8c071eb0 2024-03-27T03:14:10.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c89d9b640 10 monclient(hunting): _negotiate 2024-03-27T03:14:10.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c89d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:10.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:10.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c89d9b640 10 monclient(hunting): my global_id is 4540 2024-03-27T03:14:10.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.478+0000 7f5c89d9b640 1 -- 172.21.15.153:0/386069998 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c74003520 con 0x7f5c8c071eb0 2024-03-27T03:14:10.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 1 -- 172.21.15.153:0/386069998 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57512399 0 0) 0x7f5c8c06ef80 con 0x7f5c8c0b1f60 2024-03-27T03:14:10.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 10 monclient(hunting): _negotiate 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 10 monclient(hunting): my global_id is 4503 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 1 -- 172.21.15.153:0/386069998 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8c06f260 con 0x7f5c8c0b1f60 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 1 -- 172.21.15.153:0/386069998 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2562915397 0 0) 0x7f5c8c06f260 con 0x7f5c8c0b1f60 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 1 -- 172.21.15.153:0/386069998 >> v1:172.21.15.153:6790/0 conn(0x7f5c8c143d10 legacy=0x7f5c8c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 1 -- 172.21.15.153:0/386069998 >> v1:172.21.15.196:6789/0 conn(0x7f5c8c071eb0 legacy=0x7f5c8c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 10 monclient: _finish_hunting 0 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 1 monclient: found mon.noname-a 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 1 -- 172.21.15.153:0/386069998 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c155d70 con 0x7f5c8c0b1f60 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 10 monclient: _finish_auth 0 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 10 monclient: _check_auth_tickets 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:10.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 1 -- 172.21.15.153:0/386069998 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f5c80001670 con 0x7f5c8c0b1f60 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 10 monclient: dump: 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:10.474 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:10.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c910fb640 5 monclient: authenticate success, global_id 4503 2024-03-27T03:14:10.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c910fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:14:10.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 1 -- 172.21.15.153:0/386069998 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c800017d0 con 0x7f5c8c0b1f60 2024-03-27T03:14:10.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c89d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:10.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c910fb640 10 monclient: get_monmap_and_config success 2024-03-27T03:14:10.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.479+0000 7f5c910fb640 4 set_mon_vals no callback set 2024-03-27T03:14:10.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.480+0000 7f5c910fb640 10 monclient: shutdown 2024-03-27T03:14:10.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.480+0000 7f5c910fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:10.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.480+0000 7f5c910fb640 1 -- 172.21.15.153:0/386069998 >> v1:172.21.15.153:6789/0 conn(0x7f5c8c0b1f60 legacy=0x7f5c8c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:10.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.480+0000 7f5c910fb640 1 -- 172.21.15.153:0/386069998 shutdown_connections 2024-03-27T03:14:10.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.480+0000 7f5c910fb640 1 -- 172.21.15.153:0/386069998 >> 172.21.15.153:0/386069998 conn(0x7f5c8c066a00 msgr2=0x7f5c8c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:10.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.482+0000 7f5c910fb640 1 -- 172.21.15.153:0/386069998 shutdown_connections 2024-03-27T03:14:10.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.482+0000 7f5c910fb640 1 -- 172.21.15.153:0/386069998 wait complete. 2024-03-27T03:14:10.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.483+0000 7f5c910fb640 10 monclient: build_initial_monmap 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.483+0000 7f5c910fb640 10 monclient: monmap: 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.483+0000 7f5c910fb640 1 librados: starting msgr at 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.483+0000 7f5c910fb640 1 librados: starting objecter 2024-03-27T03:14:10.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.484+0000 7f5c910fb640 1 Processor -- start 2024-03-27T03:14:10.480 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.484+0000 7f5c910fb640 1 -- start start 2024-03-27T03:14:10.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.484+0000 7f5c910fb640 1 librados: setting wanted keys 2024-03-27T03:14:10.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.484+0000 7f5c910fb640 1 librados: calling monclient init 2024-03-27T03:14:10.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.484+0000 7f5c910fb640 10 monclient: init 2024-03-27T03:14:10.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.485+0000 7f5c910fb640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:10.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.485+0000 7f5c910fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.485+0000 7f5c910fb640 10 monclient(hunting): picked mon.noname-b con 0x7f5c8c143d10 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.486+0000 7f5c910fb640 10 monclient(hunting): picked mon.noname-c con 0x7f5c8c142e80 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.487+0000 7f5c8a59c640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f5c8c142e80 0x7f5c8c1e2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:38368/0 (socket says 172.21.15.153:38368) 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c910fb640 10 monclient(hunting): picked mon.noname-a con 0x7f5c8c145ab0 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c910fb640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c071380 con 0x7f5c8c145ab0 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c910fb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c070550 con 0x7f5c8c143d10 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c910fb640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c0715f0 con 0x7f5c8c142e80 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c910fb640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c910fb640 10 monclient(hunting): authenticate will time out at 1810.793694s 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c8a59c640 1 -- 172.21.15.153:0/4164805272 learned_addr learned my addr 172.21.15.153:0/4164805272 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c737fe640 10 client.?.objecter ms_handle_connect 0x7f5c8c142e80 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c737fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c737fe640 10 client.?.objecter ms_handle_connect 0x7f5c8c143d10 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c737fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:10.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864346155 0 0) 0x7f5c8c070550 con 0x7f5c8c143d10 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c737fe640 10 monclient(hunting): _negotiate 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c737fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c737fe640 10 monclient(hunting): my global_id is 4543 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.488+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c78004eb0 con 0x7f5c8c143d10 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.489+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3392770680 0 0) 0x7f5c78004eb0 con 0x7f5c8c143d10 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.489+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c8c070550 con 0x7f5c8c143d10 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.489+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f5c80003790 con 0x7f5c8c143d10 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.489+0000 7f5c737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.489+0000 7f5c737fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.489+0000 7f5c737fe640 10 monclient(hunting): dump: 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:10.486 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.489+0000 7f5c737fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.489+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2282103975 0 0) 0x7f5c8c070550 con 0x7f5c8c143d10 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.489+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 >> v1:172.21.15.153:6790/0 conn(0x7f5c8c142e80 legacy=0x7f5c8c1e2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 >> v1:172.21.15.153:6789/0 conn(0x7f5c8c145ab0 legacy=0x7f5c8c1e63a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c737fe640 10 monclient: _finish_hunting 0 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c737fe640 1 monclient: found mon.b 2024-03-27T03:14:10.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c737fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c1355f0 con 0x7f5c8c143d10 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c737fe640 10 monclient: _finish_auth 0 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c737fe640 10 monclient: _check_auth_tickets 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c737fe640 10 client.?.objecter ms_handle_connect 0x7f5c8c145ab0 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c737fe640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c910fb640 5 monclient: authenticate success, global_id 4543 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c910fb640 10 monclient: _renew_subs 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c910fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c910fb640 1 -- 172.21.15.153:0/4164805272 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c06f740 con 0x7f5c8c143d10 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c910fb640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c910fb640 10 monclient: _renew_subs 2024-03-27T03:14:10.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c910fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:14:10.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c910fb640 1 -- 172.21.15.153:0/4164805272 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c1e7ac0 con 0x7f5c8c143d10 2024-03-27T03:14:10.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.490+0000 7f5c910fb640 1 librados: init done 2024-03-27T03:14:10.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.492+0000 7f5c910fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:10.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.492+0000 7f5c910fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:10.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.492+0000 7f5c910fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:14:10.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.493+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c80001a80 con 0x7f5c8c143d10 2024-03-27T03:14:10.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.493+0000 7f5c910fb640 1 -- 172.21.15.153:0/4164805272 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c135780 con 0x7f5c8c143d10 2024-03-27T03:14:10.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.494+0000 7f5c737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:10.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.494+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f5c80003d40 con 0x7f5c8c143d10 2024-03-27T03:14:10.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.494+0000 7f5c737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:10.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.494+0000 7f5c737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:14:10.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.494+0000 7f5c737fe640 10 monclient: dump: 2024-03-27T03:14:10.491 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:10.491 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.494+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f5c80019d10 con 0x7f5c8c143d10 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.494+0000 7f5c8959a640 4 set_mon_vals no callback set 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.495+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f5c80083910 con 0x7f5c8c143d10 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.495+0000 7f5c737fe640 10 client.4543.objecter ms_dispatch 0x7f5c8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.495+0000 7f5c737fe640 3 client.4543.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:14:10.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.495+0000 7f5c737fe640 3 client.4543.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:14:10.493 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.495+0000 7f5c737fe640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-27T03:14:10.493 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.495+0000 7f5c8a59c640 10 monclient: get_auth_request con 0x7f5c7806dcb0 auth_method 0 2024-03-27T03:14:10.493 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.495+0000 7f5c737fe640 10 client.4543.objecter ms_handle_connect 0x7f5c7806dcb0 2024-03-27T03:14:10.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.505+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c8004f260 con 0x7f5c8c143d10 2024-03-27T03:14:10.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.505+0000 7f5c737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:10.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.505+0000 7f5c737fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:14:10.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.834+0000 7f5c910fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:10.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.834+0000 7f5c910fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:10.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.834+0000 7f5c910fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:14:10.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.834+0000 7f5c910fb640 1 -- 172.21.15.153:0/4164805272 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c8c075f10 con 0x7f5c8c143d10 2024-03-27T03:14:10.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.838+0000 7f5c737fe640 1 -- 172.21.15.153:0/4164805272 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f5c80051190 con 0x7f5c8c143d10 2024-03-27T03:14:10.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.838+0000 7f5c737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:10.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.838+0000 7f5c737fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:14:10.832 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:14:10.832 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:14:10.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.842+0000 7f5c910fb640 10 librados: watch_flush enter 2024-03-27T03:14:10.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.842+0000 7f5c910fb640 10 librados: watch_flush exit 2024-03-27T03:14:10.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.842+0000 7f5c910fb640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-27T03:14:10.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.842+0000 7f5c910fb640 10 client.4543.objecter successfully canceled tick 2024-03-27T03:14:10.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.842+0000 7f5c910fb640 1 -- 172.21.15.153:0/4164805272 >> v1:172.21.15.153:6816/2637937781 conn(0x7f5c7806dcb0 legacy=0x7f5c78070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:10.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.842+0000 7f5c910fb640 10 monclient: shutdown 2024-03-27T03:14:10.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.842+0000 7f5c910fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:10.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.842+0000 7f5c910fb640 1 -- 172.21.15.153:0/4164805272 >> v1:172.21.15.196:6789/0 conn(0x7f5c8c143d10 legacy=0x7f5c8c141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:10.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.842+0000 7f5c910fb640 1 -- 172.21.15.153:0/4164805272 shutdown_connections 2024-03-27T03:14:10.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.842+0000 7f5c910fb640 1 -- 172.21.15.153:0/4164805272 >> 172.21.15.153:0/4164805272 conn(0x7f5c8c066a00 msgr2=0x7f5c8c137af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:10.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.842+0000 7f5c910fb640 1 -- 172.21.15.153:0/4164805272 shutdown_connections 2024-03-27T03:14:10.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.844+0000 7f5c910fb640 1 -- 172.21.15.153:0/4164805272 wait complete. 2024-03-27T03:14:10.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:10.844+0000 7f5c910fb640 1 librados: shutdown 2024-03-27T03:14:10.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:14:17.662 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:14:17.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.818+0000 7fa4a8b39640 10 monclient: get_monmap_and_config 2024-03-27T03:14:17.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.818+0000 7fa4a8b39640 10 monclient: build_initial_monmap 2024-03-27T03:14:17.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.818+0000 7fa4a8b39640 10 monclient: monmap: 2024-03-27T03:14:17.811 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:17.811 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:17.812 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:14:17.819011+0000 2024-03-27T03:14:17.812 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:14:17.819011+0000 2024-03-27T03:14:17.812 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:17.812 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:17.812 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:17.812 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:14:17.812 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:14:17.812 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:17.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.819+0000 7fa4a8b39640 1 Processor -- start 2024-03-27T03:14:17.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.819+0000 7fa4a8b39640 1 -- start start 2024-03-27T03:14:17.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.819+0000 7fa4a8b39640 10 monclient: init 2024-03-27T03:14:17.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.820+0000 7fa4a8b39640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:17.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.820+0000 7fa4a8b39640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:14:17.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.820+0000 7fa4a8b39640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a41457f0 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.820+0000 7fa4a8b39640 10 monclient(hunting): picked mon.noname-c con 0x7fa4a413f650 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.820+0000 7fa4a259c640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fa4a41457f0 0x7fa4a413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:49058/0 (socket says 172.21.15.153:49058) 2024-03-27T03:14:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.820+0000 7fa4a8b39640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a40a9d70 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.820+0000 7fa4a259c640 1 -- 172.21.15.153:0/1599949872 learned_addr learned my addr 172.21.15.153:0/1599949872 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.820+0000 7fa4a8b39640 1 -- 172.21.15.153:0/1599949872 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40aaef0 con 0x7fa4a40a9d70 2024-03-27T03:14:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.820+0000 7fa4a8b39640 1 -- 172.21.15.153:0/1599949872 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a4145bd0 con 0x7fa4a413f650 2024-03-27T03:14:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.820+0000 7fa4a8b39640 1 -- 172.21.15.153:0/1599949872 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a4145eb0 con 0x7fa4a41457f0 2024-03-27T03:14:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.820+0000 7fa4a8b39640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.820+0000 7fa4a8b39640 10 monclient(hunting): authenticate will time out at 1818.126362s 2024-03-27T03:14:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.821+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203422186 0 0) 0x7fa4a4145eb0 con 0x7fa4a41457f0 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.821+0000 7fa4a159a640 10 monclient(hunting): _negotiate 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.821+0000 7fa4a159a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.821+0000 7fa4a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.821+0000 7fa4a159a640 10 monclient(hunting): my global_id is 4583 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.821+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa48c003520 con 0x7fa4a41457f0 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.821+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936400490 0 0) 0x7fa4a4145bd0 con 0x7fa4a413f650 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.821+0000 7fa4a159a640 10 monclient(hunting): _negotiate 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 10 monclient(hunting): my global_id is 4546 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a4145eb0 con 0x7fa4a413f650 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745368812 0 0) 0x7fa4a40aaef0 con 0x7fa4a40a9d70 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 10 monclient(hunting): _negotiate 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 10 monclient(hunting): my global_id is 4509 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a4145bd0 con 0x7fa4a40a9d70 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 264214771 0 0) 0x7fa48c003520 con 0x7fa4a41457f0 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 >> v1:172.21.15.196:6789/0 conn(0x7fa4a413f650 legacy=0x7fa4a40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 >> v1:172.21.15.153:6789/0 conn(0x7fa4a40a9d70 legacy=0x7fa4a414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 10 monclient: _finish_hunting 0 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 1 monclient: found mon.noname-b 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a4146340 con 0x7fa4a41457f0 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 10 monclient: _finish_auth 0 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 10 monclient: _check_auth_tickets 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.822+0000 7fa4a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.823+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa494001610 con 0x7fa4a41457f0 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.823+0000 7fa4a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.823+0000 7fa4a159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:14:17.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.823+0000 7fa4a159a640 10 monclient: dump: 2024-03-27T03:14:17.821 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:17.821 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:17.821 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:17.821 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:17.821 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:17.821 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:17.821 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:17.821 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:17.821 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:17.821 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:17.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.823+0000 7fa4a8b39640 5 monclient: authenticate success, global_id 4583 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.823+0000 7fa4a8b39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.823+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa494001770 con 0x7fa4a41457f0 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.824+0000 7fa4a159a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.824+0000 7fa4a159a640 1 -- 172.21.15.153:0/1599949872 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa494001bc0 con 0x7fa4a41457f0 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.824+0000 7fa4a8b39640 10 monclient: get_monmap_and_config success 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.824+0000 7fa4a8b39640 4 set_mon_vals no callback set 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.824+0000 7fa4a8b39640 10 monclient: shutdown 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.824+0000 7fa4a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.825+0000 7fa4a159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.825+0000 7fa4a159a640 10 monclient: dump: 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:17.822 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:17.823 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:17.823 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:17.823 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:17.823 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:17.823 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:17.823 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:17.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.825+0000 7fa4a8b39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:17.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.825+0000 7fa4a8b39640 1 -- 172.21.15.153:0/1599949872 >> v1:172.21.15.153:6790/0 conn(0x7fa4a41457f0 legacy=0x7fa4a413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:17.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.825+0000 7fa4a8b39640 1 -- 172.21.15.153:0/1599949872 shutdown_connections 2024-03-27T03:14:17.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.825+0000 7fa4a8b39640 1 -- 172.21.15.153:0/1599949872 >> 172.21.15.153:0/1599949872 conn(0x7fa4a4066a00 msgr2=0x7fa4a4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:17.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.826+0000 7fa4a8b39640 1 -- 172.21.15.153:0/1599949872 shutdown_connections 2024-03-27T03:14:17.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.826+0000 7fa4a8b39640 1 -- 172.21.15.153:0/1599949872 wait complete. 2024-03-27T03:14:17.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.828+0000 7fa4a8b39640 10 monclient: build_initial_monmap 2024-03-27T03:14:17.831 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.828+0000 7fa4a8b39640 10 monclient: monmap: 2024-03-27T03:14:17.831 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:17.831 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:14:17.831 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:14:17.831 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:14:17.831 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.828+0000 7fa4a8b39640 1 librados: starting msgr at 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.828+0000 7fa4a8b39640 1 librados: starting objecter 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.829+0000 7fa4a8b39640 1 Processor -- start 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.829+0000 7fa4a8b39640 1 -- start start 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.829+0000 7fa4a8b39640 1 librados: setting wanted keys 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.829+0000 7fa4a8b39640 1 librados: calling monclient init 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.829+0000 7fa4a8b39640 10 monclient: init 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.830+0000 7fa4a8b39640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.830+0000 7fa4a8b39640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.830+0000 7fa4a8b39640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a40a9d70 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.830+0000 7fa4a8b39640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a413f650 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.830+0000 7fa4a8b39640 10 monclient(hunting): picked mon.noname-c con 0x7fa4a41457f0 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.830+0000 7fa4a8b39640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a4142bc0 con 0x7fa4a413f650 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.830+0000 7fa4a2d9d640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fa4a41457f0 0x7fa4a41e2ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:50206/0 (socket says 172.21.15.153:50206) 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.830+0000 7fa4a2d9d640 1 -- 172.21.15.153:0/3710882399 learned_addr learned my addr 172.21.15.153:0/3710882399 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.830+0000 7fa4a8b39640 1 -- 172.21.15.153:0/3710882399 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a41cd090 con 0x7fa4a40a9d70 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.830+0000 7fa4a8b39640 1 -- 172.21.15.153:0/3710882399 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a41cb9c0 con 0x7fa4a41457f0 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.830+0000 7fa4a8b39640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa4a8b39640 10 monclient(hunting): authenticate will time out at 1818.136610s 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 client.?.objecter ms_handle_connect 0x7fa4a413f650 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 client.?.objecter ms_handle_connect 0x7fa4a41457f0 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 client.?.objecter ms_handle_connect 0x7fa4a40a9d70 2024-03-27T03:14:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778936292 0 0) 0x7fa4a4142bc0 con 0x7fa4a413f650 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 monclient(hunting): my global_id is 4512 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa484004eb0 con 0x7fa4a413f650 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848596268 0 0) 0x7fa4a41cb9c0 con 0x7fa4a41457f0 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 10 monclient(hunting): my global_id is 4586 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.831+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a4142bc0 con 0x7fa4a41457f0 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728501924 0 0) 0x7fa4a41cd090 con 0x7fa4a40a9d70 2024-03-27T03:14:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient(hunting): my global_id is 4549 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a41cb9c0 con 0x7fa4a40a9d70 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3434961888 0 0) 0x7fa484004eb0 con 0x7fa4a413f650 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4a41cd090 con 0x7fa4a413f650 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa4980019b0 con 0x7fa4a413f650 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient(hunting): dump: 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:17.835 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3549529564 0 0) 0x7fa4a4142bc0 con 0x7fa4a41457f0 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa484004eb0 con 0x7fa4a41457f0 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa49c001bf0 con 0x7fa4a41457f0 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient(hunting): dump: 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:17.836 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3403865849 0 0) 0x7fa4a41cd090 con 0x7fa4a413f650 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 >> v1:172.21.15.153:6790/0 conn(0x7fa4a41457f0 legacy=0x7fa4a41e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 >> v1:172.21.15.196:6789/0 conn(0x7fa4a40a9d70 legacy=0x7fa4a41d7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 monclient: found mon.a 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a40b1660 con 0x7fa4a413f650 2024-03-27T03:14:17.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient: _finish_auth 0 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa4a8b39640 5 monclient: authenticate success, global_id 4512 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa4a8b39640 10 monclient: _renew_subs 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa4a8b39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa498001b10 con 0x7fa4a413f650 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa4a8b39640 1 -- 172.21.15.153:0/3710882399 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a40b18b0 con 0x7fa4a413f650 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.832+0000 7fa4a8b39640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.833+0000 7fa4a8b39640 10 monclient: _renew_subs 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.833+0000 7fa4a8b39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.833+0000 7fa4a8b39640 1 -- 172.21.15.153:0/3710882399 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a41cb520 con 0x7fa4a413f650 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.833+0000 7fa4a8b39640 1 librados: init done 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.833+0000 7fa492ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.833+0000 7fa4a0d99640 4 set_mon_vals no callback set 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.833+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa498003a60 con 0x7fa4a413f650 2024-03-27T03:14:17.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.833+0000 7fa492ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.833+0000 7fa492ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.833+0000 7fa492ffd640 10 monclient: dump: 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.835+0000 7fa4a8b39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.835+0000 7fa4a8b39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:17.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.835+0000 7fa4a8b39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:14:17.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.835+0000 7fa4a8b39640 1 -- 172.21.15.153:0/3710882399 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa488005080 con 0x7fa4a413f650 2024-03-27T03:14:17.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.836+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fa498019a30 con 0x7fa4a413f650 2024-03-27T03:14:17.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.836+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fa4980838e0 con 0x7fa4a413f650 2024-03-27T03:14:17.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.836+0000 7fa492ffd640 10 client.4512.objecter ms_dispatch 0x7fa4a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:14:17.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.836+0000 7fa492ffd640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:14:17.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.836+0000 7fa492ffd640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:14:17.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.836+0000 7fa492ffd640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-27T03:14:17.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.837+0000 7fa4a259c640 10 monclient: get_auth_request con 0x7fa48406df90 auth_method 0 2024-03-27T03:14:17.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.837+0000 7fa492ffd640 10 client.4512.objecter ms_handle_connect 0x7fa48406df90 2024-03-27T03:14:17.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.844+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa49804f1c0 con 0x7fa4a413f650 2024-03-27T03:14:17.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.844+0000 7fa492ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:17.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:17.844+0000 7fa492ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:14:18.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.186+0000 7fa4a8b39640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:18.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.186+0000 7fa4a8b39640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:18.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.186+0000 7fa4a8b39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:14:18.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.186+0000 7fa4a8b39640 1 -- 172.21.15.153:0/3710882399 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa488005330 con 0x7fa4a413f650 2024-03-27T03:14:18.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.186+0000 7fa492ffd640 1 -- 172.21.15.153:0/3710882399 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fa4980510f0 con 0x7fa4a413f650 2024-03-27T03:14:18.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.186+0000 7fa492ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:18.180 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.186+0000 7fa492ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:14:18.180 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:14:18.180 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:14:18.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.189+0000 7fa4a8b39640 10 librados: watch_flush enter 2024-03-27T03:14:18.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.189+0000 7fa4a8b39640 10 librados: watch_flush exit 2024-03-27T03:14:18.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.189+0000 7fa4a8b39640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-27T03:14:18.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.189+0000 7fa4a8b39640 10 client.4512.objecter successfully canceled tick 2024-03-27T03:14:18.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.189+0000 7fa4a8b39640 1 -- 172.21.15.153:0/3710882399 >> v1:172.21.15.153:6816/2637937781 conn(0x7fa48406df90 legacy=0x7fa484070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:18.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.189+0000 7fa4a8b39640 10 monclient: shutdown 2024-03-27T03:14:18.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.189+0000 7fa4a8b39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:18.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.189+0000 7fa4a8b39640 1 -- 172.21.15.153:0/3710882399 >> v1:172.21.15.153:6789/0 conn(0x7fa4a413f650 legacy=0x7fa4a4137d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:18.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.190+0000 7fa4a8b39640 1 -- 172.21.15.153:0/3710882399 shutdown_connections 2024-03-27T03:14:18.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.190+0000 7fa4a8b39640 1 -- 172.21.15.153:0/3710882399 >> 172.21.15.153:0/3710882399 conn(0x7fa4a4066a00 msgr2=0x7fa4a4138ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:18.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.190+0000 7fa4a8b39640 1 -- 172.21.15.153:0/3710882399 shutdown_connections 2024-03-27T03:14:18.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.190+0000 7fa4a8b39640 1 -- 172.21.15.153:0/3710882399 wait complete. 2024-03-27T03:14:18.184 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:18.190+0000 7fa4a8b39640 1 librados: shutdown 2024-03-27T03:14:18.198 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:14:25.005 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:14:25.145 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.150+0000 7f6b278e2640 10 monclient: get_monmap_and_config 2024-03-27T03:14:25.145 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.150+0000 7f6b278e2640 10 monclient: build_initial_monmap 2024-03-27T03:14:25.145 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.150+0000 7f6b278e2640 10 monclient: monmap: 2024-03-27T03:14:25.145 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:25.145 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:25.145 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:14:25.152847+0000 2024-03-27T03:14:25.145 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:14:25.152847+0000 2024-03-27T03:14:25.145 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:25.145 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:25.145 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:25.146 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:14:25.146 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:14:25.146 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:25.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.151+0000 7f6b278e2640 1 Processor -- start 2024-03-27T03:14:25.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.152+0000 7f6b278e2640 1 -- start start 2024-03-27T03:14:25.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.152+0000 7f6b278e2640 10 monclient: init 2024-03-27T03:14:25.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.152+0000 7f6b278e2640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.152+0000 7f6b278e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b278e2640 10 monclient(hunting): picked mon.noname-a con 0x7f6b200b0770 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b278e2640 10 monclient(hunting): picked mon.noname-b con 0x7f6b2014a700 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b24e7d640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f6b2014a700 0x7f6b200abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:44252/0 (socket says 172.21.15.153:44252) 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b278e2640 10 monclient(hunting): picked mon.noname-c con 0x7f6b200ac5a0 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b278e2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b200aba70 con 0x7f6b200b0770 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b278e2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b200aecb0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b24e7d640 1 -- 172.21.15.153:0/3619327654 learned_addr learned my addr 172.21.15.153:0/3619327654 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b278e2640 1 -- 172.21.15.153:0/3619327654 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b200af150 con 0x7f6b2014a700 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b278e2640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b278e2640 10 monclient(hunting): authenticate will time out at 1825.460005s 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b17fff640 1 -- 172.21.15.153:0/3619327654 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621438133 0 0) 0x7f6b200af150 con 0x7f6b2014a700 2024-03-27T03:14:25.148 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b17fff640 10 monclient(hunting): _negotiate 2024-03-27T03:14:25.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b17fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:25.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:25.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b17fff640 10 monclient(hunting): my global_id is 4589 2024-03-27T03:14:25.149 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b17fff640 1 -- 172.21.15.153:0/3619327654 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b0c003520 con 0x7f6b2014a700 2024-03-27T03:14:25.150 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b17fff640 1 -- 172.21.15.153:0/3619327654 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3015692425 0 0) 0x7f6b0c003520 con 0x7f6b2014a700 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.153+0000 7f6b17fff640 1 -- 172.21.15.153:0/3619327654 >> v1:172.21.15.196:6789/0 conn(0x7f6b200ac5a0 legacy=0x7f6b20151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 1 -- 172.21.15.153:0/3619327654 >> v1:172.21.15.153:6789/0 conn(0x7f6b200b0770 legacy=0x7f6b20149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 10 monclient: _finish_hunting 0 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 1 monclient: found mon.noname-b 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 1 -- 172.21.15.153:0/3619327654 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b200ac9d0 con 0x7f6b2014a700 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 10 monclient: _finish_auth 0 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 10 monclient: _check_auth_tickets 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 1 -- 172.21.15.153:0/3619327654 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6b18001a10 con 0x7f6b2014a700 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b17fff640 10 monclient: dump: 2024-03-27T03:14:25.151 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b278e2640 5 monclient: authenticate success, global_id 4589 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.154+0000 7f6b278e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.155+0000 7f6b17fff640 1 -- 172.21.15.153:0/3619327654 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b18001be0 con 0x7f6b2014a700 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.155+0000 7f6b17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.155+0000 7f6b17fff640 1 -- 172.21.15.153:0/3619327654 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6b18001ff0 con 0x7f6b2014a700 2024-03-27T03:14:25.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.155+0000 7f6b17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.155+0000 7f6b17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.155+0000 7f6b17fff640 10 monclient: dump: 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.155+0000 7f6b278e2640 10 monclient: get_monmap_and_config success 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.155+0000 7f6b278e2640 4 set_mon_vals no callback set 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.156+0000 7f6b278e2640 10 monclient: shutdown 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.156+0000 7f6b278e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:25.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.156+0000 7f6b278e2640 1 -- 172.21.15.153:0/3619327654 >> v1:172.21.15.153:6790/0 conn(0x7f6b2014a700 legacy=0x7f6b200abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:25.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.156+0000 7f6b278e2640 1 -- 172.21.15.153:0/3619327654 shutdown_connections 2024-03-27T03:14:25.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.156+0000 7f6b278e2640 1 -- 172.21.15.153:0/3619327654 >> 172.21.15.153:0/3619327654 conn(0x7f6b2006b660 msgr2=0x7f6b2006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:25.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.157+0000 7f6b278e2640 1 -- 172.21.15.153:0/3619327654 shutdown_connections 2024-03-27T03:14:25.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.157+0000 7f6b278e2640 1 -- 172.21.15.153:0/3619327654 wait complete. 2024-03-27T03:14:25.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.158+0000 7f6b278e2640 10 monclient: build_initial_monmap 2024-03-27T03:14:25.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.158+0000 7f6b278e2640 10 monclient: monmap: 2024-03-27T03:14:25.160 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:25.160 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:14:25.160 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:14:25.160 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:14:25.160 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:25.160 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.158+0000 7f6b278e2640 1 librados: starting msgr at 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.158+0000 7f6b278e2640 1 librados: starting objecter 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.158+0000 7f6b278e2640 1 Processor -- start 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.158+0000 7f6b278e2640 1 -- start start 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.158+0000 7f6b278e2640 1 librados: setting wanted keys 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.158+0000 7f6b278e2640 1 librados: calling monclient init 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.158+0000 7f6b278e2640 10 monclient: init 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.160+0000 7f6b278e2640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.160+0000 7f6b278e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.160+0000 7f6b278e2640 10 monclient(hunting): picked mon.noname-c con 0x7f6b200ac5a0 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.160+0000 7f6b278e2640 10 monclient(hunting): picked mon.noname-a con 0x7f6b200b0770 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:25.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.160+0000 7f6b2567e640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f6b200ac5a0 0x7f6b20126560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:46062/0 (socket says 172.21.15.153:46062) 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.160+0000 7f6b2567e640 1 -- 172.21.15.153:0/1757511779 learned_addr learned my addr 172.21.15.153:0/1757511779 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b278e2640 10 monclient(hunting): picked mon.noname-b con 0x7f6b2014a700 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b20127ee0 con 0x7f6b200b0770 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b201270c0 con 0x7f6b2014a700 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b20125af0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b278e2640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b278e2640 10 monclient(hunting): authenticate will time out at 1825.467748s 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 client.?.objecter ms_handle_connect 0x7f6b200ac5a0 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 client.?.objecter ms_handle_connect 0x7f6b2014a700 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499952689 0 0) 0x7f6b20125af0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:25.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): my global_id is 4592 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b04004eb0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728573033 0 0) 0x7f6b201270c0 con 0x7f6b2014a700 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): my global_id is 4552 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b20125af0 con 0x7f6b2014a700 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 787005943 0 0) 0x7f6b04004eb0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b201270c0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6b10001960 con 0x7f6b200ac5a0 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): dump: 2024-03-27T03:14:25.163 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1414596806 0 0) 0x7f6b20125af0 con 0x7f6b2014a700 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b04004eb0 con 0x7f6b2014a700 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6b1c001490 con 0x7f6b2014a700 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.161+0000 7f6b15ffb640 10 monclient(hunting): dump: 2024-03-27T03:14:25.164 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.162+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3567223525 0 0) 0x7f6b201270c0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.162+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 >> v1:172.21.15.196:6789/0 conn(0x7f6b2014a700 legacy=0x7f6b201e5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.162+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 >> v1:172.21.15.153:6789/0 conn(0x7f6b200b0770 legacy=0x7f6b201db700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:14:25.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.165+0000 7f6b15ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:14:25.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.165+0000 7f6b15ffb640 1 monclient: found mon.c 2024-03-27T03:14:25.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.165+0000 7f6b15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:25.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.165+0000 7f6b15ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:25.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.165+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b20145900 con 0x7f6b200ac5a0 2024-03-27T03:14:25.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.165+0000 7f6b15ffb640 10 monclient: _finish_auth 0 2024-03-27T03:14:25.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.165+0000 7f6b15ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:14:25.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.165+0000 7f6b15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:25.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.165+0000 7f6b278e2640 5 monclient: authenticate success, global_id 4592 2024-03-27T03:14:25.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.165+0000 7f6b278e2640 10 monclient: _renew_subs 2024-03-27T03:14:25.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.165+0000 7f6b278e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:25.166 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.165+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b20126c70 con 0x7f6b200ac5a0 2024-03-27T03:14:25.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.166+0000 7f6b278e2640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:14:25.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.166+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b100035f0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.167 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.166+0000 7f6b15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:25.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.166+0000 7f6b177fe640 4 set_mon_vals no callback set 2024-03-27T03:14:25.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.166+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6b10003940 con 0x7f6b200ac5a0 2024-03-27T03:14:25.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.166+0000 7f6b15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:25.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.166+0000 7f6b15ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:14:25.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.166+0000 7f6b15ffb640 10 monclient: dump: 2024-03-27T03:14:25.169 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:25.169 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:25.169 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:25.169 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:25.169 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:25.169 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:25.170 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:25.170 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:25.170 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:25.170 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:25.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.166+0000 7f6b278e2640 10 monclient: _renew_subs 2024-03-27T03:14:25.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.166+0000 7f6b278e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:25.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.166+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b201dbff0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.166+0000 7f6b278e2640 1 librados: init done 2024-03-27T03:14:25.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.168+0000 7f6b278e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:25.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.168+0000 7f6b278e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:25.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.168+0000 7f6b278e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:25.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.168+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f6b10003aa0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.168+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b201dc2e0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.168+0000 7f6b24e7d640 10 monclient: get_auth_request con 0x7f6b0406dee0 auth_method 0 2024-03-27T03:14:25.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.169+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f6b100849f0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.169+0000 7f6b15ffb640 10 client.4592.objecter ms_dispatch 0x7f6b20000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:14:25.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.169+0000 7f6b15ffb640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:14:25.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.169+0000 7f6b15ffb640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:14:25.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.169+0000 7f6b15ffb640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-27T03:14:25.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.169+0000 7f6b15ffb640 10 client.4592.objecter ms_handle_connect 0x7f6b0406dee0 2024-03-27T03:14:25.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.177+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b100502d0 con 0x7f6b200ac5a0 2024-03-27T03:14:25.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.177+0000 7f6b15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:25.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.177+0000 7f6b15ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:14:25.478 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.484+0000 7f6b278e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:25.478 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.484+0000 7f6b278e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:25.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.484+0000 7f6b278e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:25.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.484+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b20075f10 con 0x7f6b200ac5a0 2024-03-27T03:14:25.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.485+0000 7f6b15ffb640 1 -- 172.21.15.153:0/1757511779 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f6b10052200 con 0x7f6b200ac5a0 2024-03-27T03:14:25.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.485+0000 7f6b15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:25.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.485+0000 7f6b15ffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:14:25.479 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:14:25.480 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:14:25.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.488+0000 7f6b278e2640 10 librados: watch_flush enter 2024-03-27T03:14:25.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.488+0000 7f6b278e2640 10 librados: watch_flush exit 2024-03-27T03:14:25.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.488+0000 7f6b278e2640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-27T03:14:25.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.488+0000 7f6b278e2640 10 client.4592.objecter successfully canceled tick 2024-03-27T03:14:25.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.488+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 >> v1:172.21.15.153:6816/2637937781 conn(0x7f6b0406dee0 legacy=0x7f6b040703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:25.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.488+0000 7f6b278e2640 10 monclient: shutdown 2024-03-27T03:14:25.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.489+0000 7f6b278e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:25.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.489+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 >> v1:172.21.15.153:6790/0 conn(0x7f6b200ac5a0 legacy=0x7f6b20126560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:25.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.489+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 shutdown_connections 2024-03-27T03:14:25.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.489+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 >> 172.21.15.153:0/1757511779 conn(0x7f6b2006b660 msgr2=0x7f6b2006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:25.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.489+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 shutdown_connections 2024-03-27T03:14:25.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.489+0000 7f6b278e2640 1 -- 172.21.15.153:0/1757511779 wait complete. 2024-03-27T03:14:25.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:25.490+0000 7f6b278e2640 1 librados: shutdown 2024-03-27T03:14:25.498 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:14:32.308 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:14:32.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.471+0000 7f688a57b640 10 monclient: get_monmap_and_config 2024-03-27T03:14:32.464 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.471+0000 7f688a57b640 10 monclient: build_initial_monmap 2024-03-27T03:14:32.465 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.471+0000 7f688a57b640 10 monclient: monmap: 2024-03-27T03:14:32.465 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:32.465 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:32.465 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:14:32.472182+0000 2024-03-27T03:14:32.465 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:14:32.472182+0000 2024-03-27T03:14:32.465 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:32.465 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:32.465 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:32.465 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:14:32.465 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:14:32.465 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:32.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.474+0000 7f688a57b640 1 Processor -- start 2024-03-27T03:14:32.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.474+0000 7f688a57b640 1 -- start start 2024-03-27T03:14:32.468 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.474+0000 7f688a57b640 10 monclient: init 2024-03-27T03:14:32.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.475+0000 7f688a57b640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:32.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.475+0000 7f688a57b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:14:32.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.475+0000 7f688a57b640 10 monclient(hunting): picked mon.noname-a con 0x7f688406d9d0 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:32.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.475+0000 7f688a57b640 10 monclient(hunting): picked mon.noname-c con 0x7f688413ef30 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:32.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.475+0000 7f688a57b640 10 monclient(hunting): picked mon.noname-b con 0x7f68840b0860 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:32.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.475+0000 7f688a57b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68840aed10 con 0x7f688406d9d0 2024-03-27T03:14:32.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.475+0000 7f688a57b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68840ab1b0 con 0x7f688413ef30 2024-03-27T03:14:32.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.475+0000 7f688a57b640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68840ab490 con 0x7f68840b0860 2024-03-27T03:14:32.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.475+0000 7f688a57b640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:32.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.475+0000 7f688a57b640 10 monclient(hunting): authenticate will time out at 1832.781455s 2024-03-27T03:14:32.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.476+0000 7f68837fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f688413ef30 0x7f68840af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:49302/0 (socket says 172.21.15.153:49302) 2024-03-27T03:14:32.469 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.476+0000 7f68837fe640 1 -- 172.21.15.153:0/1369069262 learned_addr learned my addr 172.21.15.153:0/1369069262 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:32.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.479+0000 7f6882ffd640 1 -- 172.21.15.153:0/1369069262 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019964786 0 0) 0x7f68840ab1b0 con 0x7f688413ef30 2024-03-27T03:14:32.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.479+0000 7f6882ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:14:32.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.479+0000 7f6882ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:32.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.479+0000 7f6882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:32.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.479+0000 7f6882ffd640 10 monclient(hunting): my global_id is 4555 2024-03-27T03:14:32.472 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.479+0000 7f6882ffd640 1 -- 172.21.15.153:0/1369069262 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f686c003520 con 0x7f688413ef30 2024-03-27T03:14:32.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.479+0000 7f6882ffd640 1 -- 172.21.15.153:0/1369069262 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4002153662 0 0) 0x7f686c003520 con 0x7f688413ef30 2024-03-27T03:14:32.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 1 -- 172.21.15.153:0/1369069262 >> v1:172.21.15.153:6790/0 conn(0x7f68840b0860 legacy=0x7f688414ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:14:32.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 1 -- 172.21.15.153:0/1369069262 >> v1:172.21.15.153:6789/0 conn(0x7f688406d9d0 legacy=0x7f688413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:32.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:14:32.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 1 monclient: found mon.noname-c 2024-03-27T03:14:32.473 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 1 -- 172.21.15.153:0/1369069262 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68840ab770 con 0x7f688413ef30 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 10 monclient: _finish_auth 0 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f688a57b640 5 monclient: authenticate success, global_id 4555 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f688a57b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 1 -- 172.21.15.153:0/1369069262 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6874001690 con 0x7f688413ef30 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 10 monclient: dump: 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:32.474 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f688a57b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 1 -- 172.21.15.153:0/1369069262 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6874001860 con 0x7f688413ef30 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f6882ffd640 1 -- 172.21.15.153:0/1369069262 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f6874001cb0 con 0x7f688413ef30 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f688a57b640 10 monclient: get_monmap_and_config success 2024-03-27T03:14:32.475 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.480+0000 7f688a57b640 4 set_mon_vals no callback set 2024-03-27T03:14:32.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.481+0000 7f688a57b640 10 monclient: shutdown 2024-03-27T03:14:32.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.481+0000 7f688a57b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:32.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.481+0000 7f688a57b640 1 -- 172.21.15.153:0/1369069262 >> v1:172.21.15.196:6789/0 conn(0x7f688413ef30 legacy=0x7f68840af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:32.476 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.481+0000 7f6882ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-27T03:14:32.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.483+0000 7f688a57b640 1 -- 172.21.15.153:0/1369069262 shutdown_connections 2024-03-27T03:14:32.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.483+0000 7f688a57b640 1 -- 172.21.15.153:0/1369069262 >> 172.21.15.153:0/1369069262 conn(0x7f688406b660 msgr2=0x7f688406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:32.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.484+0000 7f688a57b640 1 -- 172.21.15.153:0/1369069262 shutdown_connections 2024-03-27T03:14:32.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.484+0000 7f688a57b640 1 -- 172.21.15.153:0/1369069262 wait complete. 2024-03-27T03:14:32.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.484+0000 7f688a57b640 10 monclient: build_initial_monmap 2024-03-27T03:14:32.477 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.484+0000 7f688a57b640 10 monclient: monmap: 2024-03-27T03:14:32.478 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:32.478 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:14:32.478 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:14:32.478 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:14:32.478 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:32.478 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:32.478 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:32.478 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:14:32.478 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:14:32.478 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:32.478 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.485+0000 7f688a57b640 1 librados: starting msgr at 2024-03-27T03:14:32.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.485+0000 7f688a57b640 1 librados: starting objecter 2024-03-27T03:14:32.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.485+0000 7f688a57b640 1 Processor -- start 2024-03-27T03:14:32.482 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.485+0000 7f688a57b640 1 -- start start 2024-03-27T03:14:32.482 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.485+0000 7f688a57b640 1 librados: setting wanted keys 2024-03-27T03:14:32.482 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.485+0000 7f688a57b640 1 librados: calling monclient init 2024-03-27T03:14:32.482 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.485+0000 7f688a57b640 10 monclient: init 2024-03-27T03:14:32.482 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.486+0000 7f688a57b640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:32.482 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.486+0000 7f688a57b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:14:32.482 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.486+0000 7f688a57b640 10 monclient(hunting): picked mon.noname-a con 0x7f688406d9d0 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:32.482 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.486+0000 7f688a57b640 10 monclient(hunting): picked mon.noname-b con 0x7f68840b0860 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.488+0000 7f68837fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f68840b0860 0x7f688412f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:52640/0 (socket says 172.21.15.153:52640) 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.488+0000 7f68837fe640 1 -- 172.21.15.153:0/3610387507 learned_addr learned my addr 172.21.15.153:0/3610387507 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f688a57b640 10 monclient(hunting): picked mon.noname-c con 0x7f688413ef30 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68841c6540 con 0x7f688406d9d0 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 10 client.?.objecter ms_handle_connect 0x7f688413ef30 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688413a4e0 con 0x7f68840b0860 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6884138e30 con 0x7f688413ef30 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f688a57b640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f688a57b640 10 monclient(hunting): authenticate will time out at 1832.794872s 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 10 client.?.objecter ms_handle_connect 0x7f68840b0860 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333270285 0 0) 0x7f6884138e30 con 0x7f688413ef30 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:14:32.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:32.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:32.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 10 monclient(hunting): my global_id is 4595 2024-03-27T03:14:32.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6870004eb0 con 0x7f688413ef30 2024-03-27T03:14:32.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3928830758 0 0) 0x7f6870004eb0 con 0x7f688413ef30 2024-03-27T03:14:32.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6884138e30 con 0x7f688413ef30 2024-03-27T03:14:32.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f687c001890 con 0x7f688413ef30 2024-03-27T03:14:32.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 10 monclient(hunting): dump: 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.489+0000 7f6880ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.490+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 332201754 0 0) 0x7f6884138e30 con 0x7f688413ef30 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.490+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 >> v1:172.21.15.196:6789/0 conn(0x7f68840b0860 legacy=0x7f688412f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:32.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.490+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 >> v1:172.21.15.153:6789/0 conn(0x7f688406d9d0 legacy=0x7f6884139dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:14:32.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.492+0000 7f6880ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:14:32.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.492+0000 7f6880ff9640 1 monclient: found mon.c 2024-03-27T03:14:32.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.492+0000 7f6880ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:32.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.492+0000 7f6880ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:32.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.492+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6884138fc0 con 0x7f688413ef30 2024-03-27T03:14:32.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.492+0000 7f6880ff9640 10 monclient: _finish_auth 0 2024-03-27T03:14:32.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.492+0000 7f6880ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:14:32.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.492+0000 7f6880ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:32.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.493+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f687c001e20 con 0x7f688413ef30 2024-03-27T03:14:32.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.493+0000 7f6880ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:32.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.493+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f687c0019f0 con 0x7f688413ef30 2024-03-27T03:14:32.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.493+0000 7f6880ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:32.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.493+0000 7f6880ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:14:32.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.493+0000 7f6880ff9640 10 monclient: dump: 2024-03-27T03:14:32.488 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.494+0000 7f688a57b640 5 monclient: authenticate success, global_id 4595 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.494+0000 7f688a57b640 10 monclient: _renew_subs 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.494+0000 7f688a57b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.494+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6884139230 con 0x7f688413ef30 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.494+0000 7f688a57b640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.494+0000 7f688a57b640 10 monclient: _renew_subs 2024-03-27T03:14:32.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.494+0000 7f688a57b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:32.490 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.494+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68841300b0 con 0x7f688413ef30 2024-03-27T03:14:32.490 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.494+0000 7f688a57b640 1 librados: init done 2024-03-27T03:14:32.490 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.494+0000 7f6881ffb640 4 set_mon_vals no callback set 2024-03-27T03:14:32.490 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.496+0000 7f688a57b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:32.490 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.496+0000 7f688a57b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:32.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.496+0000 7f688a57b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:32.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.497+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f688413b9e0 con 0x7f688413ef30 2024-03-27T03:14:32.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.497+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f687c019a80 con 0x7f688413ef30 2024-03-27T03:14:32.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.497+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f687c083b40 con 0x7f688413ef30 2024-03-27T03:14:32.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.497+0000 7f6880ff9640 10 client.4595.objecter ms_dispatch 0x7f6884000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:14:32.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.497+0000 7f6880ff9640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:14:32.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.497+0000 7f6880ff9640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:14:32.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.497+0000 7f6880ff9640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-27T03:14:32.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.498+0000 7f6883fff640 10 monclient: get_auth_request con 0x7f687006dc30 auth_method 0 2024-03-27T03:14:32.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.498+0000 7f6880ff9640 10 client.4595.objecter ms_handle_connect 0x7f687006dc30 2024-03-27T03:14:32.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.506+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f687c04f250 con 0x7f688413ef30 2024-03-27T03:14:32.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.506+0000 7f6880ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:32.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.506+0000 7f6880ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:14:32.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.813+0000 7f688a57b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:32.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.813+0000 7f688a57b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:32.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.813+0000 7f688a57b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:32.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.813+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6884075f10 con 0x7f688413ef30 2024-03-27T03:14:32.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.814+0000 7f6880ff9640 1 -- 172.21.15.153:0/3610387507 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f687c051350 con 0x7f688413ef30 2024-03-27T03:14:32.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.814+0000 7f6880ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:32.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.814+0000 7f6880ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:14:32.808 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:14:32.808 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:14:32.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.817+0000 7f688a57b640 10 librados: watch_flush enter 2024-03-27T03:14:32.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.817+0000 7f688a57b640 10 librados: watch_flush exit 2024-03-27T03:14:32.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.817+0000 7f688a57b640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-27T03:14:32.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.817+0000 7f688a57b640 10 client.4595.objecter successfully canceled tick 2024-03-27T03:14:32.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.818+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 >> v1:172.21.15.153:6816/2637937781 conn(0x7f687006dc30 legacy=0x7f68700700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:32.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.818+0000 7f688a57b640 10 monclient: shutdown 2024-03-27T03:14:32.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.818+0000 7f688a57b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:32.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.818+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 >> v1:172.21.15.153:6790/0 conn(0x7f688413ef30 legacy=0x7f68841d68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:32.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.818+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 shutdown_connections 2024-03-27T03:14:32.812 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.818+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 >> 172.21.15.153:0/3610387507 conn(0x7f688406b660 msgr2=0x7f688413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:32.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.819+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 shutdown_connections 2024-03-27T03:14:32.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.819+0000 7f688a57b640 1 -- 172.21.15.153:0/3610387507 wait complete. 2024-03-27T03:14:32.813 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:32.819+0000 7f688a57b640 1 librados: shutdown 2024-03-27T03:14:32.826 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:14:39.633 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:14:39.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.798+0000 7f2603b7a640 10 monclient: get_monmap_and_config 2024-03-27T03:14:39.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.798+0000 7f2603b7a640 10 monclient: build_initial_monmap 2024-03-27T03:14:39.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.798+0000 7f2603b7a640 10 monclient: monmap: 2024-03-27T03:14:39.792 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:39.792 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:39.792 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:14:39.799773+0000 2024-03-27T03:14:39.792 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:14:39.799773+0000 2024-03-27T03:14:39.792 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:39.792 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:39.792 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:39.793 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:14:39.793 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:14:39.793 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:39.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.799+0000 7f2603b7a640 1 Processor -- start 2024-03-27T03:14:39.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.799+0000 7f2603b7a640 1 -- start start 2024-03-27T03:14:39.794 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.799+0000 7f2603b7a640 10 monclient: init 2024-03-27T03:14:39.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.800+0000 7f2603b7a640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:39.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.800+0000 7f2603b7a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:14:39.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.800+0000 7f2603b7a640 10 monclient(hunting): picked mon.noname-b con 0x7f25fc0b0770 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:39.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.800+0000 7f2603b7a640 10 monclient(hunting): picked mon.noname-c con 0x7f25fc14a700 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:39.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.800+0000 7f2603b7a640 10 monclient(hunting): picked mon.noname-a con 0x7f25fc0ac5a0 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:39.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.800+0000 7f2603b7a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc0aba70 con 0x7f25fc0ac5a0 2024-03-27T03:14:39.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.800+0000 7f2603b7a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc0aecb0 con 0x7f25fc14a700 2024-03-27T03:14:39.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.800+0000 7f2603b7a640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc0af150 con 0x7f25fc0b0770 2024-03-27T03:14:39.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.800+0000 7f2603b7a640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:39.795 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.800+0000 7f2603b7a640 10 monclient(hunting): authenticate will time out at 1840.106482s 2024-03-27T03:14:39.799 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.806+0000 7f2601916640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f25fc0b0770 0x7f25fc149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:47191/0 (socket says 172.21.15.153:47191) 2024-03-27T03:14:39.799 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.806+0000 7f2601916640 1 -- 172.21.15.153:0/297124654 learned_addr learned my addr 172.21.15.153:0/297124654 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:39.799 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 1 -- 172.21.15.153:0/297124654 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801761135 0 0) 0x7f25fc0af150 con 0x7f25fc0b0770 2024-03-27T03:14:39.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 10 monclient(hunting): _negotiate 2024-03-27T03:14:39.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:39.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:39.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 10 monclient(hunting): my global_id is 4598 2024-03-27T03:14:39.800 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 1 -- 172.21.15.153:0/297124654 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e8003520 con 0x7f25fc0b0770 2024-03-27T03:14:39.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 1 -- 172.21.15.153:0/297124654 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1846628758 0 0) 0x7f25e8003520 con 0x7f25fc0b0770 2024-03-27T03:14:39.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 1 -- 172.21.15.153:0/297124654 >> v1:172.21.15.196:6789/0 conn(0x7f25fc14a700 legacy=0x7f25fc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:39.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 1 -- 172.21.15.153:0/297124654 >> v1:172.21.15.153:6789/0 conn(0x7f25fc0ac5a0 legacy=0x7f25fc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:39.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 10 monclient: _finish_hunting 0 2024-03-27T03:14:39.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 1 monclient: found mon.noname-b 2024-03-27T03:14:39.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:39.803 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 1 -- 172.21.15.153:0/297124654 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc0ac9d0 con 0x7f25fc0b0770 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 10 monclient: _finish_auth 0 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 10 monclient: _check_auth_tickets 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 1 -- 172.21.15.153:0/297124654 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f25ec001670 con 0x7f25fc0b0770 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 10 monclient: dump: 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:39.804 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2603b7a640 5 monclient: authenticate success, global_id 4598 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2603b7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 1 -- 172.21.15.153:0/297124654 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25ec001850 con 0x7f25fc0b0770 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2600914640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2603b7a640 10 monclient: get_monmap_and_config success 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.807+0000 7f2603b7a640 4 set_mon_vals no callback set 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.808+0000 7f2603b7a640 10 monclient: shutdown 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.808+0000 7f2603b7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.808+0000 7f2603b7a640 1 -- 172.21.15.153:0/297124654 >> v1:172.21.15.153:6790/0 conn(0x7f25fc0b0770 legacy=0x7f25fc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.808+0000 7f2600914640 1 -- 172.21.15.153:0/297124654 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f25ec001670 con 0x7f25fc0b0770 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.808+0000 7f2600914640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-27T03:14:39.805 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.809+0000 7f2603b7a640 1 -- 172.21.15.153:0/297124654 shutdown_connections 2024-03-27T03:14:39.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.809+0000 7f2603b7a640 1 -- 172.21.15.153:0/297124654 >> 172.21.15.153:0/297124654 conn(0x7f25fc06b660 msgr2=0x7f25fc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:39.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.810+0000 7f2603b7a640 1 -- 172.21.15.153:0/297124654 shutdown_connections 2024-03-27T03:14:39.806 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.811+0000 7f2603b7a640 1 -- 172.21.15.153:0/297124654 wait complete. 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.812+0000 7f2603b7a640 10 monclient: build_initial_monmap 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.812+0000 7f2603b7a640 10 monclient: monmap: 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.813+0000 7f2603b7a640 1 librados: starting msgr at 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.813+0000 7f2603b7a640 1 librados: starting objecter 2024-03-27T03:14:39.807 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.813+0000 7f2603b7a640 1 Processor -- start 2024-03-27T03:14:39.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.813+0000 7f2603b7a640 1 -- start start 2024-03-27T03:14:39.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.813+0000 7f2603b7a640 1 librados: setting wanted keys 2024-03-27T03:14:39.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.813+0000 7f2603b7a640 1 librados: calling monclient init 2024-03-27T03:14:39.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.813+0000 7f2603b7a640 10 monclient: init 2024-03-27T03:14:39.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.814+0000 7f2603b7a640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:39.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.814+0000 7f2603b7a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:14:39.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.814+0000 7f2603b7a640 10 monclient(hunting): picked mon.noname-c con 0x7f25fc0ac5a0 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:39.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.814+0000 7f2603b7a640 10 monclient(hunting): picked mon.noname-b con 0x7f25fc0b0770 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:39.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.814+0000 7f2603b7a640 10 monclient(hunting): picked mon.noname-a con 0x7f25fc14a700 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:39.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.814+0000 7f2603b7a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc1290c0 con 0x7f25fc14a700 2024-03-27T03:14:39.808 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.814+0000 7f2603b7a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc127ff0 con 0x7f25fc0b0770 2024-03-27T03:14:39.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.814+0000 7f2603b7a640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc128c00 con 0x7f25fc0ac5a0 2024-03-27T03:14:39.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.814+0000 7f2603b7a640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:39.809 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.814+0000 7f2603b7a640 10 monclient(hunting): authenticate will time out at 1840.120032s 2024-03-27T03:14:39.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.817+0000 7f2601916640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f25fc0ac5a0 0x7f25fc1284f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:47269/0 (socket says 172.21.15.153:47269) 2024-03-27T03:14:39.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.817+0000 7f2601115640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f25fc0b0770 0x7f25fc1dbf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:40053/0 (socket says 172.21.15.153:40053) 2024-03-27T03:14:39.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.817+0000 7f2601115640 1 -- 172.21.15.153:0/2447554037 learned_addr learned my addr 172.21.15.153:0/2447554037 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:39.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.818+0000 7f25f27fc640 10 client.?.objecter ms_handle_connect 0x7f25fc0b0770 2024-03-27T03:14:39.811 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.818+0000 7f25f27fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:39.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.820+0000 7f25f27fc640 10 client.?.objecter ms_handle_connect 0x7f25fc0ac5a0 2024-03-27T03:14:39.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.820+0000 7f25f27fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:39.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.820+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422156713 0 0) 0x7f25fc127ff0 con 0x7f25fc0b0770 2024-03-27T03:14:39.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.820+0000 7f25f27fc640 10 monclient(hunting): _negotiate 2024-03-27T03:14:39.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.820+0000 7f25f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:39.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.820+0000 7f25f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:39.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.820+0000 7f25f27fc640 10 monclient(hunting): my global_id is 4558 2024-03-27T03:14:39.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.820+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e0004eb0 con 0x7f25fc0b0770 2024-03-27T03:14:39.819 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 10 client.?.objecter ms_handle_connect 0x7f25fc14a700 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865407343 0 0) 0x7f25fc1290c0 con 0x7f25fc14a700 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 10 monclient(hunting): _negotiate 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 10 monclient(hunting): my global_id is 4521 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25fc127ff0 con 0x7f25fc14a700 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032321708 0 0) 0x7f25fc128c00 con 0x7f25fc0ac5a0 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 10 monclient(hunting): _negotiate 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 10 monclient(hunting): my global_id is 4601 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.821+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e0005360 con 0x7f25fc0ac5a0 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.822+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 400148081 0 0) 0x7f25e0005360 con 0x7f25fc0ac5a0 2024-03-27T03:14:39.820 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.822+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25fc128c00 con 0x7f25fc0ac5a0 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.822+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1750540717 0 0) 0x7f25e0004eb0 con 0x7f25fc0b0770 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.822+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25e0005360 con 0x7f25fc0b0770 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.822+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f25ec001c20 con 0x7f25fc0ac5a0 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.822+0000 7f25f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.822+0000 7f25f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.822+0000 7f25f27fc640 10 monclient(hunting): dump: 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:39.821 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.822+0000 7f25f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f25f8001840 con 0x7f25fc0b0770 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 10 monclient(hunting): dump: 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:39.822 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 45096616 0 0) 0x7f25e0005360 con 0x7f25fc0b0770 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 >> v1:172.21.15.153:6790/0 conn(0x7f25fc0ac5a0 legacy=0x7f25fc1284f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 >> v1:172.21.15.153:6789/0 conn(0x7f25fc14a700 legacy=0x7f25fc146f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 10 monclient: _finish_hunting 0 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 1 monclient: found mon.b 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc1dc680 con 0x7f25fc0b0770 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 10 monclient: _finish_auth 0 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 10 monclient: _check_auth_tickets 2024-03-27T03:14:39.823 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f25f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:39.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f2603b7a640 5 monclient: authenticate success, global_id 4558 2024-03-27T03:14:39.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f2603b7a640 10 monclient: _renew_subs 2024-03-27T03:14:39.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f2603b7a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:14:39.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f2603b7a640 1 -- 172.21.15.153:0/2447554037 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25fc1293c0 con 0x7f25fc0b0770 2024-03-27T03:14:39.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f2603b7a640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:14:39.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f2603b7a640 10 monclient: _renew_subs 2024-03-27T03:14:39.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f2603b7a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:14:39.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f2603b7a640 1 -- 172.21.15.153:0/2447554037 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25fc1257f0 con 0x7f25fc0b0770 2024-03-27T03:14:39.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.823+0000 7f2603b7a640 1 librados: init done 2024-03-27T03:14:39.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.824+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25f80019a0 con 0x7f25fc0b0770 2024-03-27T03:14:39.824 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.824+0000 7f25f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.824+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f25f8003790 con 0x7f25fc0b0770 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.824+0000 7f25f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.824+0000 7f25f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.824+0000 7f25f27fc640 10 monclient: dump: 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:39.825 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:39.826 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:39.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.825+0000 7f25f3fff640 4 set_mon_vals no callback set 2024-03-27T03:14:39.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.826+0000 7f2603b7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:39.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.826+0000 7f2603b7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:39.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.826+0000 7f2603b7a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:14:39.826 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.826+0000 7f2603b7a640 1 -- 172.21.15.153:0/2447554037 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e4005080 con 0x7f25fc0b0770 2024-03-27T03:14:39.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.827+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f25f80038f0 con 0x7f25fc0b0770 2024-03-27T03:14:39.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.828+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f25f80848b0 con 0x7f25fc0b0770 2024-03-27T03:14:39.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.828+0000 7f25f27fc640 10 client.4558.objecter ms_dispatch 0x7f25fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:14:39.827 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.828+0000 7f25f27fc640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:14:39.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.828+0000 7f25f27fc640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:14:39.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.828+0000 7f25f27fc640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-27T03:14:39.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.828+0000 7f2601916640 10 monclient: get_auth_request con 0x7f25e006e010 auth_method 0 2024-03-27T03:14:39.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.828+0000 7f25f27fc640 10 client.4558.objecter ms_handle_connect 0x7f25e006e010 2024-03-27T03:14:39.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.834+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25f8050190 con 0x7f25fc0b0770 2024-03-27T03:14:39.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.834+0000 7f25f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:39.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:39.834+0000 7f25f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:14:40.152 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.159+0000 7f2603b7a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:40.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.159+0000 7f2603b7a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:40.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.159+0000 7f2603b7a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:14:40.153 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.159+0000 7f2603b7a640 1 -- 172.21.15.153:0/2447554037 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25e4005330 con 0x7f25fc0b0770 2024-03-27T03:14:40.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.162+0000 7f25f27fc640 1 -- 172.21.15.153:0/2447554037 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f25f80520c0 con 0x7f25fc0b0770 2024-03-27T03:14:40.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.162+0000 7f25f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:40.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.162+0000 7f25f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:14:40.155 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:14:40.155 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:14:40.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.164+0000 7f2603b7a640 10 librados: watch_flush enter 2024-03-27T03:14:40.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.165+0000 7f2603b7a640 10 librados: watch_flush exit 2024-03-27T03:14:40.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.165+0000 7f2603b7a640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-27T03:14:40.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.165+0000 7f2603b7a640 10 client.4558.objecter successfully canceled tick 2024-03-27T03:14:40.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.168+0000 7f2603b7a640 1 -- 172.21.15.153:0/2447554037 >> v1:172.21.15.153:6816/2637937781 conn(0x7f25e006e010 legacy=0x7f25e00704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:40.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.168+0000 7f2603b7a640 10 monclient: shutdown 2024-03-27T03:14:40.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.168+0000 7f2603b7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:40.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.168+0000 7f2603b7a640 1 -- 172.21.15.153:0/2447554037 >> v1:172.21.15.196:6789/0 conn(0x7f25fc0b0770 legacy=0x7f25fc1dbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:40.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.169+0000 7f2603b7a640 1 -- 172.21.15.153:0/2447554037 shutdown_connections 2024-03-27T03:14:40.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.169+0000 7f2603b7a640 1 -- 172.21.15.153:0/2447554037 >> 172.21.15.153:0/2447554037 conn(0x7f25fc06b660 msgr2=0x7f25fc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:40.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.169+0000 7f2603b7a640 1 -- 172.21.15.153:0/2447554037 shutdown_connections 2024-03-27T03:14:40.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.169+0000 7f2603b7a640 1 -- 172.21.15.153:0/2447554037 wait complete. 2024-03-27T03:14:40.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:40.169+0000 7f2603b7a640 1 librados: shutdown 2024-03-27T03:14:40.179 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:14:46.986 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:14:47.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.147+0000 7f850617b640 10 monclient: get_monmap_and_config 2024-03-27T03:14:47.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.147+0000 7f850617b640 10 monclient: build_initial_monmap 2024-03-27T03:14:47.141 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.147+0000 7f850617b640 10 monclient: monmap: 2024-03-27T03:14:47.141 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:47.141 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:47.141 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:14:47.148696+0000 2024-03-27T03:14:47.141 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:14:47.148696+0000 2024-03-27T03:14:47.141 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:47.141 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:47.141 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:47.141 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:14:47.141 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:14:47.142 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:47.146 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.151+0000 7f850617b640 1 Processor -- start 2024-03-27T03:14:47.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.151+0000 7f850617b640 1 -- start start 2024-03-27T03:14:47.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.151+0000 7f850617b640 10 monclient: init 2024-03-27T03:14:47.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.152+0000 7f850617b640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:47.147 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.152+0000 7f850617b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:14:47.150 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.154+0000 7f850617b640 10 monclient(hunting): picked mon.noname-a con 0x7f85000b1f60 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:47.150 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.154+0000 7f850617b640 10 monclient(hunting): picked mon.noname-c con 0x7f8500071eb0 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:47.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.161+0000 7f84feffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f8500071eb0 0x7f8500146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:38129/0 (socket says 172.21.15.153:38129) 2024-03-27T03:14:47.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.161+0000 7f850617b640 10 monclient(hunting): picked mon.noname-b con 0x7f8500143d10 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:47.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.161+0000 7f850617b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850006ef80 con 0x7f85000b1f60 2024-03-27T03:14:47.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.161+0000 7f850617b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850006f260 con 0x7f8500071eb0 2024-03-27T03:14:47.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.161+0000 7f850617b640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85000b40c0 con 0x7f8500143d10 2024-03-27T03:14:47.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.161+0000 7f850617b640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:47.154 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.161+0000 7f850617b640 10 monclient(hunting): authenticate will time out at 1847.467001s 2024-03-27T03:14:47.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.161+0000 7f84feffd640 1 -- 172.21.15.153:0/106368716 learned_addr learned my addr 172.21.15.153:0/106368716 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:47.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.162+0000 7f84fe7fc640 1 -- 172.21.15.153:0/106368716 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404822161 0 0) 0x7f85000b40c0 con 0x7f8500143d10 2024-03-27T03:14:47.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.162+0000 7f84fe7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:14:47.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.162+0000 7f84fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:47.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.162+0000 7f84fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:47.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.162+0000 7f84fe7fc640 10 monclient(hunting): my global_id is 4604 2024-03-27T03:14:47.155 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.162+0000 7f84fe7fc640 1 -- 172.21.15.153:0/106368716 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e8003520 con 0x7f8500143d10 2024-03-27T03:14:47.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.162+0000 7f84fe7fc640 1 -- 172.21.15.153:0/106368716 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320323729 0 0) 0x7f850006ef80 con 0x7f85000b1f60 2024-03-27T03:14:47.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.162+0000 7f84fe7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:14:47.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.162+0000 7f84fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:47.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.162+0000 7f84fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:47.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.162+0000 7f84fe7fc640 10 monclient(hunting): my global_id is 4524 2024-03-27T03:14:47.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.162+0000 7f84fe7fc640 1 -- 172.21.15.153:0/106368716 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85000b40c0 con 0x7f85000b1f60 2024-03-27T03:14:47.156 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 1 -- 172.21.15.153:0/106368716 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1901177787 0 0) 0x7f85000b40c0 con 0x7f85000b1f60 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 1 -- 172.21.15.153:0/106368716 >> v1:172.21.15.153:6790/0 conn(0x7f8500143d10 legacy=0x7f8500146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 1 -- 172.21.15.153:0/106368716 >> v1:172.21.15.196:6789/0 conn(0x7f8500071eb0 legacy=0x7f8500146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 1 monclient: found mon.noname-a 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 1 -- 172.21.15.153:0/106368716 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8500155d70 con 0x7f85000b1f60 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 10 monclient: _finish_auth 0 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 1 -- 172.21.15.153:0/106368716 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f84f00015f0 con 0x7f85000b1f60 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.163+0000 7f84fe7fc640 10 monclient: dump: 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:47.157 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:47.158 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:47.158 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:47.158 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:47.158 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:47.158 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:47.158 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:47.158 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:47.158 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:47.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.164+0000 7f850617b640 5 monclient: authenticate success, global_id 4524 2024-03-27T03:14:47.158 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.164+0000 7f850617b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.164+0000 7f84fe7fc640 1 -- 172.21.15.153:0/106368716 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84f0001750 con 0x7f85000b1f60 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.164+0000 7f84fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.164+0000 7f84fe7fc640 1 -- 172.21.15.153:0/106368716 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f84f0001b60 con 0x7f85000b1f60 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.164+0000 7f84fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.164+0000 7f84fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.164+0000 7f84fe7fc640 10 monclient: dump: 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:47.159 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:47.160 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:47.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.164+0000 7f850617b640 10 monclient: get_monmap_and_config success 2024-03-27T03:14:47.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.164+0000 7f850617b640 4 set_mon_vals no callback set 2024-03-27T03:14:47.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.165+0000 7f850617b640 10 monclient: shutdown 2024-03-27T03:14:47.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.165+0000 7f850617b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:47.160 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.165+0000 7f850617b640 1 -- 172.21.15.153:0/106368716 >> v1:172.21.15.153:6789/0 conn(0x7f85000b1f60 legacy=0x7f8500143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:47.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.166+0000 7f850617b640 1 -- 172.21.15.153:0/106368716 shutdown_connections 2024-03-27T03:14:47.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.166+0000 7f850617b640 1 -- 172.21.15.153:0/106368716 >> 172.21.15.153:0/106368716 conn(0x7f8500066a00 msgr2=0x7f8500066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:47.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.166+0000 7f850617b640 1 -- 172.21.15.153:0/106368716 shutdown_connections 2024-03-27T03:14:47.161 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.166+0000 7f850617b640 1 -- 172.21.15.153:0/106368716 wait complete. 2024-03-27T03:14:47.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.167+0000 7f850617b640 10 monclient: build_initial_monmap 2024-03-27T03:14:47.162 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.167+0000 7f850617b640 10 monclient: monmap: 2024-03-27T03:14:47.162 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:47.162 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:14:47.162 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:14:47.162 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:14:47.162 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:47.162 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:47.162 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:47.162 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:14:47.162 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:14:47.163 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:47.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.168+0000 7f850617b640 1 librados: starting msgr at 2024-03-27T03:14:47.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.168+0000 7f850617b640 1 librados: starting objecter 2024-03-27T03:14:47.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.168+0000 7f850617b640 1 Processor -- start 2024-03-27T03:14:47.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.168+0000 7f850617b640 1 -- start start 2024-03-27T03:14:47.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.168+0000 7f850617b640 1 librados: setting wanted keys 2024-03-27T03:14:47.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.168+0000 7f850617b640 1 librados: calling monclient init 2024-03-27T03:14:47.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.168+0000 7f850617b640 10 monclient: init 2024-03-27T03:14:47.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.169+0000 7f850617b640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:47.163 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.169+0000 7f850617b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:14:47.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.171+0000 7f850617b640 10 monclient(hunting): picked mon.noname-a con 0x7f85000b1f60 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:47.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.171+0000 7f850617b640 10 monclient(hunting): picked mon.noname-c con 0x7f8500143d10 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:47.164 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.171+0000 7f850617b640 10 monclient(hunting): picked mon.noname-b con 0x7f8500151ea0 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:47.165 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.171+0000 7f850617b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8500152a10 con 0x7f85000b1f60 2024-03-27T03:14:47.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.175+0000 7f84feffd640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f8500143d10 0x7f8500154d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:42179/0 (socket says 172.21.15.153:42179) 2024-03-27T03:14:47.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.175+0000 7f84feffd640 1 -- 172.21.15.153:0/2414209901 learned_addr learned my addr 172.21.15.153:0/2414209901 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:47.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.175+0000 7f84e7fff640 10 client.?.objecter ms_handle_connect 0x7f8500143d10 2024-03-27T03:14:47.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.175+0000 7f84e7fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:47.169 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.175+0000 7f850617b640 1 -- 172.21.15.153:0/2414209901 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850014b410 con 0x7f8500151ea0 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.175+0000 7f850617b640 1 -- 172.21.15.153:0/2414209901 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850014b6f0 con 0x7f8500143d10 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f850617b640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f850617b640 10 monclient(hunting): authenticate will time out at 1847.481603s 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 client.?.objecter ms_handle_connect 0x7f8500151ea0 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938871976 0 0) 0x7f850014b6f0 con 0x7f8500143d10 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 monclient(hunting): _negotiate 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 monclient(hunting): my global_id is 4607 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84ec004eb0 con 0x7f8500143d10 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 client.?.objecter ms_handle_connect 0x7f85000b1f60 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019488912 0 0) 0x7f8500152a10 con 0x7f85000b1f60 2024-03-27T03:14:47.170 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 monclient(hunting): _negotiate 2024-03-27T03:14:47.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:47.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:47.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 monclient(hunting): my global_id is 4527 2024-03-27T03:14:47.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84ec0051d0 con 0x7f85000b1f60 2024-03-27T03:14:47.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3889244129 0 0) 0x7f84ec004eb0 con 0x7f8500143d10 2024-03-27T03:14:47.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84ec005640 con 0x7f8500143d10 2024-03-27T03:14:47.171 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f84f4001930 con 0x7f8500143d10 2024-03-27T03:14:47.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:47.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:14:47.172 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 monclient(hunting): dump: 2024-03-27T03:14:47.172 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:47.172 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:47.172 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:47.172 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:47.172 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:47.172 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:47.172 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:47.172 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:47.173 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:47.173 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:47.173 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.176+0000 7f84e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:14:47.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.177+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2114877893 0 0) 0x7f84ec0051d0 con 0x7f85000b1f60 2024-03-27T03:14:47.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.177+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84ec005ce0 con 0x7f85000b1f60 2024-03-27T03:14:47.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1918144347 0 0) 0x7f84ec005640 con 0x7f8500143d10 2024-03-27T03:14:47.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 >> v1:172.21.15.196:6789/0 conn(0x7f8500151ea0 legacy=0x7f8500152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:47.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 >> v1:172.21.15.153:6789/0 conn(0x7f85000b1f60 legacy=0x7f850013fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:47.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f84e7fff640 10 monclient: _finish_hunting 0 2024-03-27T03:14:47.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f84e7fff640 1 monclient: found mon.c 2024-03-27T03:14:47.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f84e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:47.174 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f84e7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85001405e0 con 0x7f8500143d10 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f84e7fff640 10 monclient: _finish_auth 0 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f84e7fff640 10 monclient: _check_auth_tickets 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f84e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f850617b640 5 monclient: authenticate success, global_id 4607 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f850617b640 10 monclient: _renew_subs 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f850617b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.179+0000 7f850617b640 1 -- 172.21.15.153:0/2414209901 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85001defe0 con 0x7f8500143d10 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.180+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84f40047e0 con 0x7f8500143d10 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.180+0000 7f84e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.180+0000 7f84fdffb640 4 set_mon_vals no callback set 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.180+0000 7f850617b640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.180+0000 7f850617b640 10 monclient: _renew_subs 2024-03-27T03:14:47.175 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.180+0000 7f850617b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.180+0000 7f850617b640 1 -- 172.21.15.153:0/2414209901 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85001df4c0 con 0x7f8500143d10 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.180+0000 7f850617b640 1 librados: init done 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.180+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f84f4001930 con 0x7f8500143d10 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.181+0000 7f84e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.181+0000 7f84e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.181+0000 7f84e7fff640 10 monclient: dump: 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:47.176 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:47.177 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:47.177 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:47.177 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:47.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.183+0000 7f850617b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:47.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.183+0000 7f850617b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:47.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.183+0000 7f850617b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:47.177 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.183+0000 7f850617b640 1 -- 172.21.15.153:0/2414209901 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8500075f10 con 0x7f8500143d10 2024-03-27T03:14:47.178 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.184+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f84f401a880 con 0x7f8500143d10 2024-03-27T03:14:47.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.185+0000 7f84ff7fe640 10 monclient: get_auth_request con 0x7f84ec06e520 auth_method 0 2024-03-27T03:14:47.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.186+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f84f40544c0 con 0x7f8500143d10 2024-03-27T03:14:47.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.186+0000 7f84e7fff640 10 client.4607.objecter ms_dispatch 0x7f8500000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:14:47.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.186+0000 7f84e7fff640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:14:47.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.186+0000 7f84e7fff640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:14:47.179 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.186+0000 7f84e7fff640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-27T03:14:47.183 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.190+0000 7f84e7fff640 10 client.4607.objecter ms_handle_connect 0x7f84ec06e520 2024-03-27T03:14:47.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.194+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f84f4050760 con 0x7f8500143d10 2024-03-27T03:14:47.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.194+0000 7f84e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:47.187 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.194+0000 7f84e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:14:47.509 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.513+0000 7f850617b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:47.509 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.513+0000 7f850617b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:47.509 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.513+0000 7f850617b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:47.509 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.513+0000 7f850617b640 1 -- 172.21.15.153:0/2414209901 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8500140770 con 0x7f8500143d10 2024-03-27T03:14:47.509 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.516+0000 7f84e7fff640 1 -- 172.21.15.153:0/2414209901 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f84f4050760 con 0x7f8500143d10 2024-03-27T03:14:47.509 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.516+0000 7f84e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:47.509 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.516+0000 7f84e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:14:47.510 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:14:47.510 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:14:47.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.520+0000 7f850617b640 10 librados: watch_flush enter 2024-03-27T03:14:47.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.520+0000 7f850617b640 10 librados: watch_flush exit 2024-03-27T03:14:47.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.520+0000 7f850617b640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-27T03:14:47.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.520+0000 7f850617b640 10 client.4607.objecter successfully canceled tick 2024-03-27T03:14:47.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.520+0000 7f850617b640 1 -- 172.21.15.153:0/2414209901 >> v1:172.21.15.153:6816/2637937781 conn(0x7f84ec06e520 legacy=0x7f84ec0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:47.513 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.520+0000 7f850617b640 10 monclient: shutdown 2024-03-27T03:14:47.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.520+0000 7f850617b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:47.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.520+0000 7f850617b640 1 -- 172.21.15.153:0/2414209901 >> v1:172.21.15.153:6790/0 conn(0x7f8500143d10 legacy=0x7f8500154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:47.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.520+0000 7f850617b640 1 -- 172.21.15.153:0/2414209901 shutdown_connections 2024-03-27T03:14:47.514 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.520+0000 7f850617b640 1 -- 172.21.15.153:0/2414209901 >> 172.21.15.153:0/2414209901 conn(0x7f8500066a00 msgr2=0x7f85000b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:47.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.523+0000 7f850617b640 1 -- 172.21.15.153:0/2414209901 shutdown_connections 2024-03-27T03:14:47.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.523+0000 7f850617b640 1 -- 172.21.15.153:0/2414209901 wait complete. 2024-03-27T03:14:47.516 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:47.523+0000 7f850617b640 1 librados: shutdown 2024-03-27T03:14:47.533 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:14:54.340 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:14:54.479 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.486+0000 7f439a27b640 10 monclient: get_monmap_and_config 2024-03-27T03:14:54.480 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.486+0000 7f439a27b640 10 monclient: build_initial_monmap 2024-03-27T03:14:54.480 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.486+0000 7f439a27b640 10 monclient: monmap: 2024-03-27T03:14:54.480 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:54.480 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:54.480 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:14:54.487278+0000 2024-03-27T03:14:54.480 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:14:54.487278+0000 2024-03-27T03:14:54.480 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:54.480 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:54.480 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:54.480 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:14:54.480 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:14:54.480 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:54.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.487+0000 7f439a27b640 1 Processor -- start 2024-03-27T03:14:54.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.487+0000 7f439a27b640 1 -- start start 2024-03-27T03:14:54.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.487+0000 7f439a27b640 10 monclient: init 2024-03-27T03:14:54.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.488+0000 7f439a27b640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:54.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.488+0000 7f439a27b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:14:54.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.488+0000 7f439a27b640 10 monclient(hunting): picked mon.noname-a con 0x7f43940b0860 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:54.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f439a27b640 10 monclient(hunting): picked mon.noname-b con 0x7f439413ef30 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:54.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f439a27b640 10 monclient(hunting): picked mon.noname-c con 0x7f43940af8e0 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:54.483 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f439a27b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43940b0c40 con 0x7f43940b0860 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f439a27b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43940aed10 con 0x7f43940af8e0 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f43937fe640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f439413ef30 0x7f43940af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:38430/0 (socket says 172.21.15.153:38430) 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f43937fe640 1 -- 172.21.15.153:0/4262735107 learned_addr learned my addr 172.21.15.153:0/4262735107 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f439a27b640 1 -- 172.21.15.153:0/4262735107 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43940afd50 con 0x7f439413ef30 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f439a27b640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f439a27b640 10 monclient(hunting): authenticate will time out at 1854.794987s 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f4392ffd640 1 -- 172.21.15.153:0/4262735107 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429337548 0 0) 0x7f43940b0c40 con 0x7f43940b0860 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f4392ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f4392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f4392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f4392ffd640 10 monclient(hunting): my global_id is 4530 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f4392ffd640 1 -- 172.21.15.153:0/4262735107 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f437c003db0 con 0x7f43940b0860 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f4392ffd640 1 -- 172.21.15.153:0/4262735107 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2531550483 0 0) 0x7f437c003db0 con 0x7f43940b0860 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f4392ffd640 1 -- 172.21.15.153:0/4262735107 >> v1:172.21.15.153:6790/0 conn(0x7f439413ef30 legacy=0x7f43940af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:54.484 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.489+0000 7f4392ffd640 1 -- 172.21.15.153:0/4262735107 >> v1:172.21.15.196:6789/0 conn(0x7f43940af8e0 legacy=0x7f4394149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:14:54.485 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 1 monclient: found mon.noname-a 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 1 -- 172.21.15.153:0/4262735107 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43940ab1b0 con 0x7f43940b0860 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 10 monclient: _finish_auth 0 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 1 -- 172.21.15.153:0/4262735107 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4388001670 con 0x7f43940b0860 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 10 monclient: dump: 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:54.486 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f439a27b640 5 monclient: authenticate success, global_id 4530 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f439a27b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 1 -- 172.21.15.153:0/4262735107 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4388001900 con 0x7f43940b0860 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f4392ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f439a27b640 10 monclient: get_monmap_and_config success 2024-03-27T03:14:54.487 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.492+0000 7f439a27b640 4 set_mon_vals no callback set 2024-03-27T03:14:54.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.493+0000 7f4392ffd640 1 -- 172.21.15.153:0/4262735107 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4388001cd0 con 0x7f43940b0860 2024-03-27T03:14:54.488 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.493+0000 7f4392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.493+0000 7f4392ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.493+0000 7f4392ffd640 10 monclient: dump: 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.493+0000 7f439a27b640 10 monclient: shutdown 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.493+0000 7f439a27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.493+0000 7f439a27b640 1 -- 172.21.15.153:0/4262735107 >> v1:172.21.15.153:6789/0 conn(0x7f43940b0860 legacy=0x7f439413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:54.489 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.493+0000 7f439a27b640 1 -- 172.21.15.153:0/4262735107 shutdown_connections 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.493+0000 7f439a27b640 1 -- 172.21.15.153:0/4262735107 >> 172.21.15.153:0/4262735107 conn(0x7f439406b660 msgr2=0x7f439406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.494+0000 7f439a27b640 1 -- 172.21.15.153:0/4262735107 shutdown_connections 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.494+0000 7f439a27b640 1 -- 172.21.15.153:0/4262735107 wait complete. 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.495+0000 7f439a27b640 10 monclient: build_initial_monmap 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.495+0000 7f439a27b640 10 monclient: monmap: 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:14:54.490 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:54.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.496+0000 7f439a27b640 1 librados: starting msgr at 2024-03-27T03:14:54.491 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.496+0000 7f439a27b640 1 librados: starting objecter 2024-03-27T03:14:54.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.497+0000 7f439a27b640 1 Processor -- start 2024-03-27T03:14:54.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.497+0000 7f439a27b640 1 -- start start 2024-03-27T03:14:54.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.497+0000 7f439a27b640 1 librados: setting wanted keys 2024-03-27T03:14:54.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.497+0000 7f439a27b640 1 librados: calling monclient init 2024-03-27T03:14:54.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.497+0000 7f439a27b640 10 monclient: init 2024-03-27T03:14:54.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.498+0000 7f439a27b640 10 monclient: _reopen_session rank -1 2024-03-27T03:14:54.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.498+0000 7f439a27b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:14:54.492 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.498+0000 7f439a27b640 10 monclient(hunting): picked mon.noname-b con 0x7f43940af8e0 addr v1:172.21.15.196:6789/0 2024-03-27T03:14:54.493 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.500+0000 7f439a27b640 10 monclient(hunting): picked mon.noname-a con 0x7f43940b0860 addr v1:172.21.15.153:6789/0 2024-03-27T03:14:54.493 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.500+0000 7f43937fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f43940b0860 0x7f439412f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:32972/0 (socket says 172.21.15.153:32972) 2024-03-27T03:14:54.495 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.500+0000 7f4398818640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f439413ef30 0x7f439412fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:42858/0 (socket says 172.21.15.153:42858) 2024-03-27T03:14:54.495 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.500+0000 7f4398818640 1 -- 172.21.15.153:0/3484796032 learned_addr learned my addr 172.21.15.153:0/3484796032 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:14:54.495 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.500+0000 7f4390ff9640 10 client.?.objecter ms_handle_connect 0x7f439413ef30 2024-03-27T03:14:54.495 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.500+0000 7f4390ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:54.495 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 client.?.objecter ms_handle_connect 0x7f43940b0860 2024-03-27T03:14:54.495 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:14:54.495 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f439a27b640 10 monclient(hunting): picked mon.noname-c con 0x7f439413ef30 addr v1:172.21.15.153:6790/0 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4394138e30 con 0x7f43940b0860 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4394130120 con 0x7f43940af8e0 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439413aae0 con 0x7f439413ef30 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f439a27b640 10 monclient(hunting): _renew_subs 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f439a27b640 10 monclient(hunting): authenticate will time out at 1854.806779s 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208975005 0 0) 0x7f439413aae0 con 0x7f439413ef30 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): my global_id is 4613 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4380004eb0 con 0x7f439413ef30 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597852380 0 0) 0x7f4394138e30 con 0x7f43940b0860 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:14:54.496 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): my global_id is 4533 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f439413aae0 con 0x7f43940b0860 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3844156195 0 0) 0x7f4380004eb0 con 0x7f439413ef30 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4394138e30 con 0x7f439413ef30 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f438c001b70 con 0x7f439413ef30 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): dump: 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:54.497 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2113474179 0 0) 0x7f439413aae0 con 0x7f43940b0860 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4380004eb0 con 0x7f43940b0860 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f4384001850 con 0x7f43940b0860 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 10 monclient(hunting): dump: 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:54.498 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2738037186 0 0) 0x7f4394138e30 con 0x7f439413ef30 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.501+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 >> v1:172.21.15.196:6789/0 conn(0x7f43940af8e0 legacy=0x7f43941392c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 >> v1:172.21.15.153:6789/0 conn(0x7f43940b0860 legacy=0x7f439412f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 1 monclient: found mon.c 2024-03-27T03:14:54.499 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439413adc0 con 0x7f439413ef30 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 10 monclient: _finish_auth 0 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f439a27b640 5 monclient: authenticate success, global_id 4613 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f439a27b640 10 monclient: _renew_subs 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f439a27b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43941c8220 con 0x7f439413ef30 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f439a27b640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f439a27b640 10 monclient: _renew_subs 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f439a27b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43941d6b80 con 0x7f439413ef30 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f439a27b640 1 librados: init done 2024-03-27T03:14:54.500 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f438c002080 con 0x7f439413ef30 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f438c004e20 con 0x7f439413ef30 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f43927fc640 4 set_mon_vals no callback set 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.502+0000 7f4390ff9640 10 monclient: dump: 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:14:54.501 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:14:54.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.504+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f438c01adf0 con 0x7f439413ef30 2024-03-27T03:14:54.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.505+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f438c084da0 con 0x7f439413ef30 2024-03-27T03:14:54.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.505+0000 7f4390ff9640 10 client.4613.objecter ms_dispatch 0x7f4394000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:14:54.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.505+0000 7f4390ff9640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:14:54.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.505+0000 7f4390ff9640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:14:54.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.505+0000 7f4393fff640 10 monclient: get_auth_request con 0x7f438006ddb0 auth_method 0 2024-03-27T03:14:54.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.505+0000 7f4390ff9640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-27T03:14:54.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.505+0000 7f439a27b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:54.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.505+0000 7f439a27b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:54.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.505+0000 7f439a27b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:54.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.505+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4374005080 con 0x7f439413ef30 2024-03-27T03:14:54.503 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.505+0000 7f4390ff9640 10 client.4613.objecter ms_handle_connect 0x7f438006ddb0 2024-03-27T03:14:54.506 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.513+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f438c0504b0 con 0x7f439413ef30 2024-03-27T03:14:54.506 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.513+0000 7f4390ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:14:54.506 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.513+0000 7f4390ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:14:54.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.834+0000 7f439a27b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:54.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.834+0000 7f439a27b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:54.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.834+0000 7f439a27b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:14:54.828 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.834+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4374005330 con 0x7f439413ef30 2024-03-27T03:14:54.829 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:14:54.829 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:14:54.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.836+0000 7f4390ff9640 1 -- 172.21.15.153:0/3484796032 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f438c0525b0 con 0x7f439413ef30 2024-03-27T03:14:54.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.836+0000 7f4390ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:14:54.829 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.836+0000 7f4390ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:14:54.832 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.839+0000 7f439a27b640 10 librados: watch_flush enter 2024-03-27T03:14:54.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.840+0000 7f439a27b640 10 librados: watch_flush exit 2024-03-27T03:14:54.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.840+0000 7f439a27b640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-27T03:14:54.833 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.840+0000 7f439a27b640 10 client.4613.objecter successfully canceled tick 2024-03-27T03:14:54.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.840+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 >> v1:172.21.15.153:6816/2637937781 conn(0x7f438006ddb0 legacy=0x7f4380070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:54.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.840+0000 7f439a27b640 10 monclient: shutdown 2024-03-27T03:14:54.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.840+0000 7f439a27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:14:54.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.840+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 >> v1:172.21.15.153:6790/0 conn(0x7f439413ef30 legacy=0x7f439412fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:14:54.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.840+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 shutdown_connections 2024-03-27T03:14:54.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.840+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 >> 172.21.15.153:0/3484796032 conn(0x7f439406b660 msgr2=0x7f439413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:14:54.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.842+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 shutdown_connections 2024-03-27T03:14:54.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.843+0000 7f439a27b640 1 -- 172.21.15.153:0/3484796032 wait complete. 2024-03-27T03:14:54.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:14:54.843+0000 7f439a27b640 1 librados: shutdown 2024-03-27T03:14:54.854 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:15:01.661 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:15:01.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.840+0000 7f2627271640 10 monclient: get_monmap_and_config 2024-03-27T03:15:01.834 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.841+0000 7f2627271640 10 monclient: build_initial_monmap 2024-03-27T03:15:01.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.841+0000 7f2627271640 10 monclient: monmap: 2024-03-27T03:15:01.835 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:01.835 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:01.835 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:15:01.841940+0000 2024-03-27T03:15:01.835 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:15:01.841940+0000 2024-03-27T03:15:01.835 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:01.835 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:01.835 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:01.835 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:15:01.835 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:15:01.835 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:01.835 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.841+0000 7f2627271640 1 Processor -- start 2024-03-27T03:15:01.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.842+0000 7f2627271640 1 -- start start 2024-03-27T03:15:01.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.842+0000 7f2627271640 10 monclient: init 2024-03-27T03:15:01.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.843+0000 7f2627271640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:01.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.843+0000 7f2627271640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:15:01.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.843+0000 7f2627271640 10 monclient(hunting): picked mon.noname-b con 0x7f26200acfa0 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:01.836 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.843+0000 7f2627271640 10 monclient(hunting): picked mon.noname-c con 0x7f2620072f10 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:01.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.843+0000 7f262626f640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f26200acfa0 0x7f2620141450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:37282/0 (socket says 172.21.15.153:37282) 2024-03-27T03:15:01.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.843+0000 7f2627271640 10 monclient(hunting): picked mon.noname-a con 0x7f26200ad380 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:01.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.843+0000 7f2627271640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26200aa9f0 con 0x7f26200ad380 2024-03-27T03:15:01.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.843+0000 7f2627271640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26200ac4c0 con 0x7f2620072f10 2024-03-27T03:15:01.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.843+0000 7f2627271640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26200ac7a0 con 0x7f26200acfa0 2024-03-27T03:15:01.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.843+0000 7f2627271640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:01.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.843+0000 7f2627271640 10 monclient(hunting): authenticate will time out at 1862.149179s 2024-03-27T03:15:01.837 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.843+0000 7f262626f640 1 -- 172.21.15.153:0/3180373890 learned_addr learned my addr 172.21.15.153:0/3180373890 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:01.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291934455 0 0) 0x7f26200ac7a0 con 0x7f26200acfa0 2024-03-27T03:15:01.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 10 monclient(hunting): _negotiate 2024-03-27T03:15:01.838 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 10 monclient(hunting): my global_id is 4616 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f260c003520 con 0x7f26200acfa0 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629348472 0 0) 0x7f26200aa9f0 con 0x7f26200ad380 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 10 monclient(hunting): _negotiate 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 10 monclient(hunting): my global_id is 4536 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26200ac7a0 con 0x7f26200ad380 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790553815 0 0) 0x7f26200ac4c0 con 0x7f2620072f10 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 10 monclient(hunting): _negotiate 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:01.839 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 10 monclient(hunting): my global_id is 4564 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.844+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26200aa9f0 con 0x7f2620072f10 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 460819534 0 0) 0x7f260c003520 con 0x7f26200acfa0 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 >> v1:172.21.15.196:6789/0 conn(0x7f2620072f10 legacy=0x7f2620141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 >> v1:172.21.15.153:6789/0 conn(0x7f26200ad380 legacy=0x7f26201497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 10 monclient: _finish_hunting 0 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 1 monclient: found mon.noname-b 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26200ab5a0 con 0x7f26200acfa0 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 10 monclient: _finish_auth 0 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 10 monclient: _check_auth_tickets 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2610001690 con 0x7f26200acfa0 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:15:01.840 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 10 monclient: dump: 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f2627271640 5 monclient: authenticate success, global_id 4616 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f2627271640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26100018b0 con 0x7f26200acfa0 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 1 -- 172.21.15.153:0/3180373890 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2610001c40 con 0x7f26200acfa0 2024-03-27T03:15:01.841 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:01.842 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:15:01.842 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f262526d640 10 monclient: dump: 2024-03-27T03:15:01.842 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:01.842 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:01.842 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:01.842 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:01.842 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:01.842 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:01.842 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:01.842 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:01.842 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:01.842 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:01.846 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f2627271640 10 monclient: get_monmap_and_config success 2024-03-27T03:15:01.846 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.845+0000 7f2627271640 4 set_mon_vals no callback set 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.846+0000 7f2627271640 10 monclient: shutdown 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.846+0000 7f2627271640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.846+0000 7f2627271640 1 -- 172.21.15.153:0/3180373890 >> v1:172.21.15.153:6790/0 conn(0x7f26200acfa0 legacy=0x7f2620141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.847+0000 7f2627271640 1 -- 172.21.15.153:0/3180373890 shutdown_connections 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.847+0000 7f2627271640 1 -- 172.21.15.153:0/3180373890 >> 172.21.15.153:0/3180373890 conn(0x7f26200b3a30 msgr2=0x7f26200b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.848+0000 7f2627271640 1 -- 172.21.15.153:0/3180373890 shutdown_connections 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.848+0000 7f2627271640 1 -- 172.21.15.153:0/3180373890 wait complete. 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.848+0000 7f2627271640 10 monclient: build_initial_monmap 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.848+0000 7f2627271640 10 monclient: monmap: 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:01.847 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.849+0000 7f2627271640 1 librados: starting msgr at 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.849+0000 7f2627271640 1 librados: starting objecter 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.849+0000 7f2627271640 1 Processor -- start 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.849+0000 7f2627271640 1 -- start start 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.849+0000 7f2627271640 1 librados: setting wanted keys 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.849+0000 7f2627271640 1 librados: calling monclient init 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.849+0000 7f2627271640 10 monclient: init 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.850+0000 7f2627271640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.850+0000 7f2627271640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.850+0000 7f2627271640 10 monclient(hunting): picked mon.noname-b con 0x7f2620072f10 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.850+0000 7f2627271640 10 monclient(hunting): picked mon.noname-c con 0x7f26200acfa0 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:01.848 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.850+0000 7f2625a6e640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f26200acfa0 0x7f2620136240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:37662/0 (socket says 172.21.15.153:37662) 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.850+0000 7f2625a6e640 1 -- 172.21.15.153:0/390896166 learned_addr learned my addr 172.21.15.153:0/390896166 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.850+0000 7f2627271640 10 monclient(hunting): picked mon.noname-a con 0x7f26200ad380 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.850+0000 7f2616ffd640 10 client.?.objecter ms_handle_connect 0x7f26200ad380 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.850+0000 7f2616ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.850+0000 7f2616ffd640 10 client.?.objecter ms_handle_connect 0x7f2620072f10 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.850+0000 7f2616ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.850+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2620137060 con 0x7f26200ad380 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 10 client.?.objecter ms_handle_connect 0x7f26200acfa0 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2620137320 con 0x7f2620072f10 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26201c8fc0 con 0x7f26200acfa0 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2627271640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2627271640 10 monclient(hunting): authenticate will time out at 1862.156809s 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557741735 0 0) 0x7f2620137320 con 0x7f2620072f10 2024-03-27T03:15:01.849 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 10 monclient(hunting): my global_id is 4567 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2604004eb0 con 0x7f2620072f10 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015832585 0 0) 0x7f26201c8fc0 con 0x7f26200acfa0 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 10 monclient(hunting): my global_id is 4619 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2620137320 con 0x7f26200acfa0 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.851+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535350313 0 0) 0x7f2620137060 con 0x7f26200ad380 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:01.850 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 10 monclient(hunting): my global_id is 4539 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26201c8fc0 con 0x7f26200ad380 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1466409638 0 0) 0x7f2604004eb0 con 0x7f2620072f10 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2620137060 con 0x7f2620072f10 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f26100015d0 con 0x7f2620072f10 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 10 monclient(hunting): dump: 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:01.851 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2818148021 0 0) 0x7f2620137320 con 0x7f26200acfa0 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2604004eb0 con 0x7f26200acfa0 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f261c001960 con 0x7f26200acfa0 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.852+0000 7f2616ffd640 10 monclient(hunting): dump: 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:01.852 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2426035368 0 0) 0x7f2620137060 con 0x7f2620072f10 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 >> v1:172.21.15.153:6790/0 conn(0x7f26200acfa0 legacy=0x7f2620136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 >> v1:172.21.15.153:6789/0 conn(0x7f26200ad380 legacy=0x7f2620136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 1 monclient: found mon.b 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26201c80f0 con 0x7f2620072f10 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 10 monclient: _finish_auth 0 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:15:01.853 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2627271640 5 monclient: authenticate success, global_id 4567 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2627271640 10 monclient: _renew_subs 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2627271640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26201c94c0 con 0x7f2620072f10 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2627271640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2627271640 10 monclient: _renew_subs 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2627271640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26201dbc40 con 0x7f2620072f10 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2627271640 1 librados: init done 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2610004a90 con 0x7f2620072f10 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f2610004da0 con 0x7f2620072f10 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:01.854 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:15:01.855 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2624a6c640 4 set_mon_vals no callback set 2024-03-27T03:15:01.855 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.853+0000 7f2616ffd640 10 monclient: dump: 2024-03-27T03:15:01.855 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:01.855 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:01.855 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:01.855 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:01.855 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:01.855 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:01.855 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:01.855 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:01.855 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:01.855 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:01.856 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.854+0000 7f2627271640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.854+0000 7f2627271640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.854+0000 7f2627271640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.854+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2608005080 con 0x7f2620072f10 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.856+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f261001ad70 con 0x7f2620072f10 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.856+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f2610084b00 con 0x7f2620072f10 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.856+0000 7f2616ffd640 10 client.4567.objecter ms_dispatch 0x7f2620000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.856+0000 7f2616ffd640 3 client.4567.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.856+0000 7f2616ffd640 3 client.4567.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.856+0000 7f2616ffd640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.856+0000 7f2625a6e640 10 monclient: get_auth_request con 0x7f260406df90 auth_method 0 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.857+0000 7f2616ffd640 10 client.4567.objecter ms_handle_connect 0x7f260406df90 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.862+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26100503e0 con 0x7f2620072f10 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.862+0000 7f2616ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:01.857 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:01.862+0000 7f2616ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:15:02.215 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.222+0000 7f2627271640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:02.215 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.222+0000 7f2627271640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:02.215 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.222+0000 7f2627271640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:15:02.216 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.222+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2608005330 con 0x7f2620072f10 2024-03-27T03:15:02.216 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.223+0000 7f2616ffd640 1 -- 172.21.15.153:0/390896166 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f2610052310 con 0x7f2620072f10 2024-03-27T03:15:02.216 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.223+0000 7f2616ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:02.216 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.223+0000 7f2616ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:15:02.216 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:15:02.216 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:15:02.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.226+0000 7f2627271640 10 librados: watch_flush enter 2024-03-27T03:15:02.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.226+0000 7f2627271640 10 librados: watch_flush exit 2024-03-27T03:15:02.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.226+0000 7f2627271640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-27T03:15:02.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.227+0000 7f2627271640 10 client.4567.objecter successfully canceled tick 2024-03-27T03:15:02.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.227+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 >> v1:172.21.15.153:6816/2637937781 conn(0x7f260406df90 legacy=0x7f2604070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:02.220 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.227+0000 7f2627271640 10 monclient: shutdown 2024-03-27T03:15:02.221 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.227+0000 7f2627271640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:02.221 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.227+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 >> v1:172.21.15.196:6789/0 conn(0x7f2620072f10 legacy=0x7f26201db350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:02.221 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.227+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 shutdown_connections 2024-03-27T03:15:02.221 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.227+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 >> 172.21.15.153:0/390896166 conn(0x7f26200b3a30 msgr2=0x7f26200b3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:02.221 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.228+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 shutdown_connections 2024-03-27T03:15:02.222 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.229+0000 7f2627271640 1 -- 172.21.15.153:0/390896166 wait complete. 2024-03-27T03:15:02.222 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:02.229+0000 7f2627271640 1 librados: shutdown 2024-03-27T03:15:02.237 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:15:09.045 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:15:09.916 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.923+0000 7fd62dae6640 10 monclient: get_monmap_and_config 2024-03-27T03:15:09.917 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.923+0000 7fd62dae6640 10 monclient: build_initial_monmap 2024-03-27T03:15:09.917 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.923+0000 7fd62dae6640 10 monclient: monmap: 2024-03-27T03:15:09.917 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:09.917 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:09.917 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:15:09.924432+0000 2024-03-27T03:15:09.917 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:15:09.924432+0000 2024-03-27T03:15:09.917 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:09.917 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:09.917 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:09.917 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:15:09.917 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:15:09.918 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:09.919 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.924+0000 7fd62dae6640 1 Processor -- start 2024-03-27T03:15:09.919 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.924+0000 7fd62dae6640 1 -- start start 2024-03-27T03:15:09.919 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.924+0000 7fd62dae6640 10 monclient: init 2024-03-27T03:15:09.919 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.925+0000 7fd62dae6640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:09.919 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.925+0000 7fd62dae6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:15:09.919 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.925+0000 7fd62dae6640 10 monclient(hunting): picked mon.noname-a con 0x7fd6280b0770 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:09.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.925+0000 7fd62dae6640 10 monclient(hunting): picked mon.noname-c con 0x7fd62814a700 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:09.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.925+0000 7fd62dae6640 10 monclient(hunting): picked mon.noname-b con 0x7fd6280ac5a0 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:09.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.925+0000 7fd62dae6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6280aba70 con 0x7fd6280b0770 2024-03-27T03:15:09.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.925+0000 7fd62dae6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6280aecb0 con 0x7fd62814a700 2024-03-27T03:15:09.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.925+0000 7fd62dae6640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6280af150 con 0x7fd6280ac5a0 2024-03-27T03:15:09.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.925+0000 7fd62dae6640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:09.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.925+0000 7fd62dae6640 10 monclient(hunting): authenticate will time out at 1870.231558s 2024-03-27T03:15:09.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.926+0000 7fd627fff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fd6280ac5a0 0x7fd628151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:41474/0 (socket says 172.21.15.153:41474) 2024-03-27T03:15:09.920 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.926+0000 7fd627fff640 1 -- 172.21.15.153:0/3030600286 learned_addr learned my addr 172.21.15.153:0/3030600286 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:09.921 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 1 -- 172.21.15.153:0/3030600286 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854974869 0 0) 0x7fd6280af150 con 0x7fd6280ac5a0 2024-03-27T03:15:09.921 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient(hunting): _negotiate 2024-03-27T03:15:09.921 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:09.921 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:09.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient(hunting): my global_id is 4622 2024-03-27T03:15:09.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 1 -- 172.21.15.153:0/3030600286 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd610003520 con 0x7fd6280ac5a0 2024-03-27T03:15:09.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 1 -- 172.21.15.153:0/3030600286 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458713477 0 0) 0x7fd6280aecb0 con 0x7fd62814a700 2024-03-27T03:15:09.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient(hunting): _negotiate 2024-03-27T03:15:09.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:09.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:09.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient(hunting): my global_id is 4570 2024-03-27T03:15:09.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 1 -- 172.21.15.153:0/3030600286 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6280af150 con 0x7fd62814a700 2024-03-27T03:15:09.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 1 -- 172.21.15.153:0/3030600286 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2344952096 0 0) 0x7fd610003520 con 0x7fd6280ac5a0 2024-03-27T03:15:09.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 1 -- 172.21.15.153:0/3030600286 >> v1:172.21.15.196:6789/0 conn(0x7fd62814a700 legacy=0x7fd6280abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:09.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 1 -- 172.21.15.153:0/3030600286 >> v1:172.21.15.153:6789/0 conn(0x7fd6280b0770 legacy=0x7fd628149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:09.922 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient: _finish_hunting 0 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 1 monclient: found mon.noname-b 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 1 -- 172.21.15.153:0/3030600286 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6280ac9d0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient: _finish_auth 0 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient: _check_auth_tickets 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd62dae6640 5 monclient: authenticate success, global_id 4622 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd62dae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 1 -- 172.21.15.153:0/3030600286 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd618001690 con 0x7fd6280ac5a0 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd6267fc640 10 monclient: dump: 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:09.923 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.927+0000 7fd62dae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.928+0000 7fd6267fc640 1 -- 172.21.15.153:0/3030600286 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6180017f0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.928+0000 7fd6267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.928+0000 7fd62dae6640 10 monclient: get_monmap_and_config success 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.928+0000 7fd62dae6640 4 set_mon_vals no callback set 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.928+0000 7fd6267fc640 1 -- 172.21.15.153:0/3030600286 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd618001c00 con 0x7fd6280ac5a0 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.928+0000 7fd6267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.928+0000 7fd6267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:15:09.924 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.928+0000 7fd6267fc640 10 monclient: dump: 2024-03-27T03:15:09.925 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:09.925 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:09.925 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:09.925 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:09.925 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:09.925 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:09.925 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:09.925 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:09.925 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:09.925 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:09.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.929+0000 7fd62dae6640 10 monclient: shutdown 2024-03-27T03:15:09.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.929+0000 7fd62dae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:09.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.929+0000 7fd62dae6640 1 -- 172.21.15.153:0/3030600286 >> v1:172.21.15.153:6790/0 conn(0x7fd6280ac5a0 legacy=0x7fd628151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:09.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.929+0000 7fd62dae6640 1 -- 172.21.15.153:0/3030600286 shutdown_connections 2024-03-27T03:15:09.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.929+0000 7fd62dae6640 1 -- 172.21.15.153:0/3030600286 >> 172.21.15.153:0/3030600286 conn(0x7fd62806b660 msgr2=0x7fd62806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:09.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.929+0000 7fd62dae6640 1 -- 172.21.15.153:0/3030600286 shutdown_connections 2024-03-27T03:15:09.933 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.929+0000 7fd62dae6640 1 -- 172.21.15.153:0/3030600286 wait complete. 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.930+0000 7fd62dae6640 10 monclient: build_initial_monmap 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.930+0000 7fd62dae6640 10 monclient: monmap: 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.931+0000 7fd62dae6640 1 librados: starting msgr at 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.931+0000 7fd62dae6640 1 librados: starting objecter 2024-03-27T03:15:09.934 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.931+0000 7fd62dae6640 1 Processor -- start 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.931+0000 7fd62dae6640 1 -- start start 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.931+0000 7fd62dae6640 1 librados: setting wanted keys 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.931+0000 7fd62dae6640 1 librados: calling monclient init 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.931+0000 7fd62dae6640 10 monclient: init 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.932+0000 7fd62dae6640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.932+0000 7fd62dae6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.933+0000 7fd62dae6640 10 monclient(hunting): picked mon.noname-b con 0x7fd6280ac5a0 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.933+0000 7fd62dae6640 10 monclient(hunting): picked mon.noname-a con 0x7fd6280b0770 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.933+0000 7fd62dae6640 10 monclient(hunting): picked mon.noname-c con 0x7fd62814a700 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.933+0000 7fd62dae6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd62812a0f0 con 0x7fd6280b0770 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.933+0000 7fd6277fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd6280ac5a0 0x7fd6281deaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:48690/0 (socket says 172.21.15.153:48690) 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.933+0000 7fd6277fe640 1 -- 172.21.15.153:0/15495106 learned_addr learned my addr 172.21.15.153:0/15495106 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.933+0000 7fd626ffd640 1 --1- 172.21.15.153:0/15495106 >> v1:172.21.15.153:6789/0 conn(0x7fd6280b0770 0x7fd628146fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:35866/0 (socket says 172.21.15.153:35866) 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.933+0000 7fd62dae6640 1 -- 172.21.15.153:0/15495106 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6281289f0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.935 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.933+0000 7fd60ffff640 10 client.?.objecter ms_handle_connect 0x7fd6280ac5a0 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.933+0000 7fd60ffff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.933+0000 7fd60ffff640 10 client.?.objecter ms_handle_connect 0x7fd6280b0770 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.933+0000 7fd60ffff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.934+0000 7fd62dae6640 1 -- 172.21.15.153:0/15495106 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd62812a320 con 0x7fd62814a700 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.934+0000 7fd62dae6640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.934+0000 7fd62dae6640 10 monclient(hunting): authenticate will time out at 1870.239710s 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.934+0000 7fd60ffff640 10 client.?.objecter ms_handle_connect 0x7fd62814a700 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.934+0000 7fd60ffff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.934+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305177356 0 0) 0x7fd6281289f0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.934+0000 7fd60ffff640 10 monclient(hunting): _negotiate 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.934+0000 7fd60ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.934+0000 7fd60ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.934+0000 7fd60ffff640 10 monclient(hunting): my global_id is 4573 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.934+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd614004eb0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881621830 0 0) 0x7fd62812a320 con 0x7fd62814a700 2024-03-27T03:15:09.936 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 10 monclient(hunting): _negotiate 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 10 monclient(hunting): my global_id is 4625 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6281289f0 con 0x7fd62814a700 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018347124 0 0) 0x7fd62812a0f0 con 0x7fd6280b0770 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 10 monclient(hunting): _negotiate 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 10 monclient(hunting): my global_id is 4545 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd62812a320 con 0x7fd6280b0770 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2984772440 0 0) 0x7fd614004eb0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd62812a0f0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd620001d50 con 0x7fd6280ac5a0 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.935+0000 7fd60ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:09.937 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.936+0000 7fd60ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.936+0000 7fd60ffff640 10 monclient(hunting): dump: 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.936+0000 7fd60ffff640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.936+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4132825431 0 0) 0x7fd6281289f0 con 0x7fd62814a700 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.936+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd614004eb0 con 0x7fd62814a700 2024-03-27T03:15:09.938 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.936+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd618001b00 con 0x7fd62814a700 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.936+0000 7fd60ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.936+0000 7fd60ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.936+0000 7fd60ffff640 10 monclient(hunting): dump: 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.936+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2963487723 0 0) 0x7fd62812a0f0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.939 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.936+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 >> v1:172.21.15.153:6790/0 conn(0x7fd62814a700 legacy=0x7fd6281e6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.936+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 >> v1:172.21.15.153:6789/0 conn(0x7fd6280b0770 legacy=0x7fd628146fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 10 monclient: _finish_hunting 0 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 1 monclient: found mon.b 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6281df200 con 0x7fd6280ac5a0 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 10 monclient: _finish_auth 0 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 10 monclient: _check_auth_tickets 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd62dae6640 5 monclient: authenticate success, global_id 4573 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd62dae6640 10 monclient: _renew_subs 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd62dae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd62dae6640 1 -- 172.21.15.153:0/15495106 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6281d90b0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd62dae6640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd62dae6640 10 monclient: _renew_subs 2024-03-27T03:15:09.940 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd62dae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd62dae6640 1 -- 172.21.15.153:0/15495106 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6281478c0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd62dae6640 1 librados: init done 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd620001eb0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd620003c40 con 0x7fd6280ac5a0 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd60ffff640 10 monclient: dump: 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:09.941 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:09.942 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:09.942 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:09.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.937+0000 7fd625ffb640 4 set_mon_vals no callback set 2024-03-27T03:15:09.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.938+0000 7fd62dae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:09.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.938+0000 7fd62dae6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:09.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.938+0000 7fd62dae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:15:09.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.939+0000 7fd62dae6640 1 -- 172.21.15.153:0/15495106 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd608005080 con 0x7fd6280ac5a0 2024-03-27T03:15:09.942 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.940+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fd620003da0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.941+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fd620084dd0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.943 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.941+0000 7fd60ffff640 10 client.4573.objecter ms_dispatch 0x7fd628000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:15:09.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.941+0000 7fd60ffff640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:15:09.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.941+0000 7fd60ffff640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:15:09.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.941+0000 7fd60ffff640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-27T03:15:09.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.941+0000 7fd626ffd640 10 monclient: get_auth_request con 0x7fd61406df50 auth_method 0 2024-03-27T03:15:09.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.941+0000 7fd60ffff640 10 client.4573.objecter ms_handle_connect 0x7fd61406df50 2024-03-27T03:15:09.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.947+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd6200504e0 con 0x7fd6280ac5a0 2024-03-27T03:15:09.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.947+0000 7fd60ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:09.944 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:09.947+0000 7fd60ffff640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:15:10.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.319+0000 7fd62dae6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:10.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.319+0000 7fd62dae6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:10.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.319+0000 7fd62dae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:15:10.313 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.319+0000 7fd62dae6640 1 -- 172.21.15.153:0/15495106 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd608005330 con 0x7fd6280ac5a0 2024-03-27T03:15:10.313 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.320+0000 7fd60ffff640 1 -- 172.21.15.153:0/15495106 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fd6200525e0 con 0x7fd6280ac5a0 2024-03-27T03:15:10.313 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.320+0000 7fd60ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:10.313 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.320+0000 7fd60ffff640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:15:10.314 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:15:10.314 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:15:10.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.324+0000 7fd62dae6640 10 librados: watch_flush enter 2024-03-27T03:15:10.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.324+0000 7fd62dae6640 10 librados: watch_flush exit 2024-03-27T03:15:10.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.324+0000 7fd62dae6640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-27T03:15:10.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.324+0000 7fd62dae6640 10 client.4573.objecter successfully canceled tick 2024-03-27T03:15:10.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.324+0000 7fd62dae6640 1 -- 172.21.15.153:0/15495106 >> v1:172.21.15.153:6816/2637937781 conn(0x7fd61406df50 legacy=0x7fd614070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:10.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.324+0000 7fd62dae6640 10 monclient: shutdown 2024-03-27T03:15:10.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.325+0000 7fd62dae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:10.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.325+0000 7fd62dae6640 1 -- 172.21.15.153:0/15495106 >> v1:172.21.15.196:6789/0 conn(0x7fd6280ac5a0 legacy=0x7fd6281deaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:10.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.325+0000 7fd62dae6640 1 -- 172.21.15.153:0/15495106 shutdown_connections 2024-03-27T03:15:10.318 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.325+0000 7fd62dae6640 1 -- 172.21.15.153:0/15495106 >> 172.21.15.153:0/15495106 conn(0x7fd62806b660 msgr2=0x7fd6280b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:10.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.325+0000 7fd62dae6640 1 -- 172.21.15.153:0/15495106 shutdown_connections 2024-03-27T03:15:10.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.325+0000 7fd62dae6640 1 -- 172.21.15.153:0/15495106 wait complete. 2024-03-27T03:15:10.319 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:10.325+0000 7fd62dae6640 1 librados: shutdown 2024-03-27T03:15:10.334 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:15:17.140 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:15:17.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.294+0000 7fa5baafb640 10 monclient: get_monmap_and_config 2024-03-27T03:15:17.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.294+0000 7fa5baafb640 10 monclient: build_initial_monmap 2024-03-27T03:15:17.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.294+0000 7fa5baafb640 10 monclient: monmap: 2024-03-27T03:15:17.290 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:17.290 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:17.290 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:15:17.295654+0000 2024-03-27T03:15:17.290 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:15:17.295654+0000 2024-03-27T03:15:17.290 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:17.290 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:17.290 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:17.290 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:15:17.290 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:15:17.290 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:17.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.296+0000 7fa5baafb640 1 Processor -- start 2024-03-27T03:15:17.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.296+0000 7fa5baafb640 1 -- start start 2024-03-27T03:15:17.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.296+0000 7fa5baafb640 10 monclient: init 2024-03-27T03:15:17.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.297+0000 7fa5baafb640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:17.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.297+0000 7fa5baafb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:15:17.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.297+0000 7fa5baafb640 10 monclient(hunting): picked mon.noname-b con 0x7fa5b40a9ac0 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:17.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.297+0000 7fa5baafb640 10 monclient(hunting): picked mon.noname-a con 0x7fa5b413ee90 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:17.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.297+0000 7fa5baafb640 10 monclient(hunting): picked mon.noname-c con 0x7fa5b40af900 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:17.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.297+0000 7fa5baafb640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b40afd10 con 0x7fa5b413ee90 2024-03-27T03:15:17.293 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.297+0000 7fa5baafb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b40af6f0 con 0x7fa5b40af900 2024-03-27T03:15:17.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.297+0000 7fa5baafb640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b40aac80 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.297+0000 7fa5baafb640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:17.294 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.297+0000 7fa5baafb640 10 monclient(hunting): authenticate will time out at 1877.603223s 2024-03-27T03:15:17.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.301+0000 7fa5b8897640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fa5b40a9ac0 0x7fa5b40adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:41927/0 (socket says 172.21.15.153:41927) 2024-03-27T03:15:17.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.301+0000 7fa5b8897640 1 -- 172.21.15.153:0/294510979 learned_addr learned my addr 172.21.15.153:0/294510979 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:17.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.301+0000 7fa5b37fe640 1 -- 172.21.15.153:0/294510979 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67745437 0 0) 0x7fa5b40aac80 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.301+0000 7fa5b37fe640 10 monclient(hunting): _negotiate 2024-03-27T03:15:17.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.301+0000 7fa5b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:17.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.301+0000 7fa5b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:17.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.301+0000 7fa5b37fe640 10 monclient(hunting): my global_id is 4628 2024-03-27T03:15:17.295 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.301+0000 7fa5b37fe640 1 -- 172.21.15.153:0/294510979 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa59c003520 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 1 -- 172.21.15.153:0/294510979 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1245502951 0 0) 0x7fa59c003520 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 1 -- 172.21.15.153:0/294510979 >> v1:172.21.15.196:6789/0 conn(0x7fa5b40af900 legacy=0x7fa5b4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:15:17.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 1 -- 172.21.15.153:0/294510979 >> v1:172.21.15.153:6789/0 conn(0x7fa5b413ee90 legacy=0x7fa5b413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:15:17.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 10 monclient: _finish_hunting 0 2024-03-27T03:15:17.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 1 monclient: found mon.noname-b 2024-03-27T03:15:17.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:17.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:15:17.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 1 -- 172.21.15.153:0/294510979 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b40aaf60 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 10 monclient: _finish_auth 0 2024-03-27T03:15:17.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 10 monclient: _check_auth_tickets 2024-03-27T03:15:17.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:17.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 1 -- 172.21.15.153:0/294510979 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa5a4001690 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:17.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.303+0000 7fa5b37fe640 10 monclient: dump: 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.304+0000 7fa5baafb640 5 monclient: authenticate success, global_id 4628 2024-03-27T03:15:17.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.304+0000 7fa5baafb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:15:17.301 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.304+0000 7fa5b37fe640 1 -- 172.21.15.153:0/294510979 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5a4001860 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.301 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.304+0000 7fa5b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:17.301 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.304+0000 7fa5b37fe640 1 -- 172.21.15.153:0/294510979 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa5a4001cb0 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.301 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.304+0000 7fa5b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:17.301 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.304+0000 7fa5b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:15:17.301 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.304+0000 7fa5b37fe640 10 monclient: dump: 2024-03-27T03:15:17.301 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:17.301 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:17.301 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:17.301 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:17.301 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:17.302 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:17.302 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:17.302 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:17.302 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:17.302 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:17.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.304+0000 7fa5baafb640 10 monclient: get_monmap_and_config success 2024-03-27T03:15:17.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.304+0000 7fa5baafb640 4 set_mon_vals no callback set 2024-03-27T03:15:17.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.305+0000 7fa5baafb640 10 monclient: shutdown 2024-03-27T03:15:17.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.305+0000 7fa5baafb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:17.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.305+0000 7fa5baafb640 1 -- 172.21.15.153:0/294510979 >> v1:172.21.15.153:6790/0 conn(0x7fa5b40a9ac0 legacy=0x7fa5b40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:17.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.305+0000 7fa5baafb640 1 -- 172.21.15.153:0/294510979 shutdown_connections 2024-03-27T03:15:17.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.305+0000 7fa5baafb640 1 -- 172.21.15.153:0/294510979 >> 172.21.15.153:0/294510979 conn(0x7fa5b40b35e0 msgr2=0x7fa5b40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:17.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.305+0000 7fa5baafb640 1 -- 172.21.15.153:0/294510979 shutdown_connections 2024-03-27T03:15:17.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.305+0000 7fa5baafb640 1 -- 172.21.15.153:0/294510979 wait complete. 2024-03-27T03:15:17.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.306+0000 7fa5baafb640 10 monclient: build_initial_monmap 2024-03-27T03:15:17.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.306+0000 7fa5baafb640 10 monclient: monmap: 2024-03-27T03:15:17.305 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:17.305 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:15:17.305 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:15:17.305 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:15:17.306 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:17.306 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:17.306 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:17.306 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:15:17.306 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:15:17.306 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:17.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.307+0000 7fa5baafb640 1 librados: starting msgr at 2024-03-27T03:15:17.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.307+0000 7fa5baafb640 1 librados: starting objecter 2024-03-27T03:15:17.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.307+0000 7fa5baafb640 1 Processor -- start 2024-03-27T03:15:17.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.307+0000 7fa5baafb640 1 -- start start 2024-03-27T03:15:17.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.307+0000 7fa5baafb640 1 librados: setting wanted keys 2024-03-27T03:15:17.306 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.307+0000 7fa5baafb640 1 librados: calling monclient init 2024-03-27T03:15:17.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.307+0000 7fa5baafb640 10 monclient: init 2024-03-27T03:15:17.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.308+0000 7fa5baafb640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:17.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.308+0000 7fa5baafb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:15:17.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.308+0000 7fa5baafb640 10 monclient(hunting): picked mon.noname-a con 0x7fa5b40a9ac0 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:17.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.308+0000 7fa5baafb640 10 monclient(hunting): picked mon.noname-c con 0x7fa5b40af900 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:17.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.308+0000 7fa5baafb640 10 monclient(hunting): picked mon.noname-b con 0x7fa5b413ee90 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:17.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.308+0000 7fa5baafb640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b41425e0 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.308+0000 7fa5baafb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b41df7a0 con 0x7fa5b413ee90 2024-03-27T03:15:17.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.308+0000 7fa5baafb640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b41416f0 con 0x7fa5b40af900 2024-03-27T03:15:17.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.308+0000 7fa5baafb640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:17.307 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.308+0000 7fa5baafb640 10 monclient(hunting): authenticate will time out at 1877.614162s 2024-03-27T03:15:17.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.314+0000 7fa5b9098640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fa5b413ee90 0x7fa5b41df090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:40489/0 (socket says 172.21.15.153:40489) 2024-03-27T03:15:17.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.314+0000 7fa5b9098640 1 -- 172.21.15.153:0/1596459316 learned_addr learned my addr 172.21.15.153:0/1596459316 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:17.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.316+0000 7fa5b17fa640 10 client.?.objecter ms_handle_connect 0x7fa5b40af900 2024-03-27T03:15:17.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.316+0000 7fa5b17fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:17.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.316+0000 7fa5b17fa640 10 client.?.objecter ms_handle_connect 0x7fa5b40a9ac0 2024-03-27T03:15:17.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.316+0000 7fa5b17fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:17.311 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.316+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170697548 0 0) 0x7fa5b41416f0 con 0x7fa5b40af900 2024-03-27T03:15:17.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.317+0000 7fa5b17fa640 10 monclient(hunting): _negotiate 2024-03-27T03:15:17.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.317+0000 7fa5b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:17.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.317+0000 7fa5b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:17.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.317+0000 7fa5b17fa640 10 monclient(hunting): my global_id is 4631 2024-03-27T03:15:17.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.317+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a0004eb0 con 0x7fa5b40af900 2024-03-27T03:15:17.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.317+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371841884 0 0) 0x7fa5b41425e0 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.317+0000 7fa5b17fa640 10 monclient(hunting): _negotiate 2024-03-27T03:15:17.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.317+0000 7fa5b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:17.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.317+0000 7fa5b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:17.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.317+0000 7fa5b17fa640 10 monclient(hunting): my global_id is 4548 2024-03-27T03:15:17.312 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.317+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b41416f0 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.319+0000 7fa5b17fa640 10 client.?.objecter ms_handle_connect 0x7fa5b413ee90 2024-03-27T03:15:17.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.319+0000 7fa5b17fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:17.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.319+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822124784 0 0) 0x7fa5b41df7a0 con 0x7fa5b413ee90 2024-03-27T03:15:17.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.319+0000 7fa5b17fa640 10 monclient(hunting): _negotiate 2024-03-27T03:15:17.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.319+0000 7fa5b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:17.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.320+0000 7fa5b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:17.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.320+0000 7fa5b17fa640 10 monclient(hunting): my global_id is 4576 2024-03-27T03:15:17.320 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.320+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b41425e0 con 0x7fa5b413ee90 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.320+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2707815355 0 0) 0x7fa5b41416f0 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.320+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5b41df7a0 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.320+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 143793977 0 0) 0x7fa5a0004eb0 con 0x7fa5b40af900 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.320+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5b41416f0 con 0x7fa5b40af900 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.320+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa5a4001b70 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.320+0000 7fa5b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.320+0000 7fa5b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.320+0000 7fa5b17fa640 10 monclient(hunting): dump: 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:17.321 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.321+0000 7fa5b17fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.321+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa5a80017f0 con 0x7fa5b40af900 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.321+0000 7fa5b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.321+0000 7fa5b17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.323+0000 7fa5b17fa640 10 monclient(hunting): dump: 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:17.322 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1162848168 0 0) 0x7fa5b41425e0 con 0x7fa5b413ee90 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5a0004eb0 con 0x7fa5b413ee90 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4255594725 0 0) 0x7fa5b41df7a0 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 >> v1:172.21.15.153:6790/0 conn(0x7fa5b40af900 legacy=0x7fa5b41de980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 >> v1:172.21.15.196:6789/0 conn(0x7fa5b413ee90 legacy=0x7fa5b41df090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 10 monclient: _finish_hunting 0 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 1 monclient: found mon.a 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b41c8160 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 10 monclient: _finish_auth 0 2024-03-27T03:15:17.323 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 10 monclient: _check_auth_tickets 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5baafb640 5 monclient: authenticate success, global_id 4548 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5baafb640 10 monclient: _renew_subs 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5baafb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5baafb640 1 -- 172.21.15.153:0/1596459316 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b41c8440 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5baafb640 10 client.4548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5baafb640 10 monclient: _renew_subs 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5baafb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5baafb640 1 -- 172.21.15.153:0/1596459316 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b41c8a00 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5baafb640 1 librados: init done 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5a4004a90 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fa5a4004da0 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:17.324 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b17fa640 10 monclient: dump: 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.324+0000 7fa5b2ffd640 4 set_mon_vals no callback set 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.326+0000 7fa5baafb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.326+0000 7fa5baafb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.326+0000 7fa5baafb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:17.325 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.326+0000 7fa5baafb640 1 -- 172.21.15.153:0/1596459316 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa594005080 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.326 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.326+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fa5a401ad70 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.326 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.327+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7fa5a40849a0 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.327 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.327+0000 7fa5b3fff640 10 monclient: get_auth_request con 0x7fa5a006e020 auth_method 0 2024-03-27T03:15:17.327 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.328+0000 7fa5b17fa640 10 client.4548.objecter ms_dispatch 0x7fa5b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:15:17.327 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.328+0000 7fa5b17fa640 3 client.4548.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:15:17.327 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.328+0000 7fa5b17fa640 3 client.4548.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:15:17.327 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.328+0000 7fa5b17fa640 20 client.4548.objecter dump_active .. 0 homeless 2024-03-27T03:15:17.327 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.328+0000 7fa5b17fa640 10 client.4548.objecter ms_handle_connect 0x7fa5a006e020 2024-03-27T03:15:17.329 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.336+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5a4050280 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.331 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.338+0000 7fa5b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:17.331 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.338+0000 7fa5b17fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:15:17.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.661+0000 7fa5baafb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:17.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.661+0000 7fa5baafb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:17.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.661+0000 7fa5baafb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:17.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.661+0000 7fa5baafb640 1 -- 172.21.15.153:0/1596459316 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa594005330 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.661+0000 7fa5b17fa640 1 -- 172.21.15.153:0/1596459316 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7fa5a40521b0 con 0x7fa5b40a9ac0 2024-03-27T03:15:17.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.662+0000 7fa5b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:17.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.662+0000 7fa5b17fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:15:17.656 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:15:17.656 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:15:17.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.666+0000 7fa5baafb640 10 librados: watch_flush enter 2024-03-27T03:15:17.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.667+0000 7fa5baafb640 10 librados: watch_flush exit 2024-03-27T03:15:17.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.667+0000 7fa5baafb640 20 client.4548.objecter shutdown clearing up homeless session... 2024-03-27T03:15:17.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.667+0000 7fa5baafb640 10 client.4548.objecter successfully canceled tick 2024-03-27T03:15:17.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.667+0000 7fa5baafb640 1 -- 172.21.15.153:0/1596459316 >> v1:172.21.15.153:6816/2637937781 conn(0x7fa5a006e020 legacy=0x7fa5a00704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:17.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.667+0000 7fa5baafb640 10 monclient: shutdown 2024-03-27T03:15:17.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.667+0000 7fa5baafb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:17.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.667+0000 7fa5baafb640 1 -- 172.21.15.153:0/1596459316 >> v1:172.21.15.153:6789/0 conn(0x7fa5b40a9ac0 legacy=0x7fa5b4141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:17.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.670+0000 7fa5baafb640 1 -- 172.21.15.153:0/1596459316 shutdown_connections 2024-03-27T03:15:17.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.670+0000 7fa5baafb640 1 -- 172.21.15.153:0/1596459316 >> 172.21.15.153:0/1596459316 conn(0x7fa5b40b35e0 msgr2=0x7fa5b413b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:17.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.671+0000 7fa5baafb640 1 -- 172.21.15.153:0/1596459316 shutdown_connections 2024-03-27T03:15:17.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.671+0000 7fa5baafb640 1 -- 172.21.15.153:0/1596459316 wait complete. 2024-03-27T03:15:17.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:17.671+0000 7fa5baafb640 1 librados: shutdown 2024-03-27T03:15:17.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:15:24.489 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:15:24.640 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.646+0000 7ff555201640 10 monclient: get_monmap_and_config 2024-03-27T03:15:24.640 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.646+0000 7ff555201640 10 monclient: build_initial_monmap 2024-03-27T03:15:24.640 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.646+0000 7ff555201640 10 monclient: monmap: 2024-03-27T03:15:24.640 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:24.640 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:24.640 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:15:24.648115+0000 2024-03-27T03:15:24.641 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:15:24.648115+0000 2024-03-27T03:15:24.641 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:24.641 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:24.641 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:24.641 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:15:24.641 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:15:24.641 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:24.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.649+0000 7ff555201640 1 Processor -- start 2024-03-27T03:15:24.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.649+0000 7ff555201640 1 -- start start 2024-03-27T03:15:24.643 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.649+0000 7ff555201640 10 monclient: init 2024-03-27T03:15:24.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.650+0000 7ff555201640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:24.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.650+0000 7ff555201640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:15:24.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.650+0000 7ff555201640 10 monclient(hunting): picked mon.noname-b con 0x7ff55006fb00 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:24.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.650+0000 7ff555201640 10 monclient(hunting): picked mon.noname-a con 0x7ff5500b0420 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:24.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.650+0000 7ff555201640 10 monclient(hunting): picked mon.noname-c con 0x7ff5500af990 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:24.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.650+0000 7ff555201640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff550140c10 con 0x7ff5500b0420 2024-03-27T03:15:24.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.650+0000 7ff555201640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff550071fa0 con 0x7ff5500af990 2024-03-27T03:15:24.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.650+0000 7ff555201640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff550072280 con 0x7ff55006fb00 2024-03-27T03:15:24.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.650+0000 7ff555201640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:24.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.650+0000 7ff555201640 10 monclient(hunting): authenticate will time out at 1884.956846s 2024-03-27T03:15:24.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.654+0000 7ff554a00640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ff5500af990 0x7ff55014b6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:38883/0 (socket says 172.21.15.153:38883) 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.654+0000 7ff554a00640 1 -- 172.21.15.153:0/1550645332 learned_addr learned my addr 172.21.15.153:0/1550645332 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 1 -- 172.21.15.153:0/1550645332 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002681360 0 0) 0x7ff550071fa0 con 0x7ff5500af990 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 10 monclient(hunting): _negotiate 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 10 monclient(hunting): my global_id is 4579 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 1 -- 172.21.15.153:0/1550645332 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff534003520 con 0x7ff5500af990 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 1 -- 172.21.15.153:0/1550645332 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911152275 0 0) 0x7ff550072280 con 0x7ff55006fb00 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 10 monclient(hunting): _negotiate 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 10 monclient(hunting): my global_id is 4634 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 1 -- 172.21.15.153:0/1550645332 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff550071fa0 con 0x7ff55006fb00 2024-03-27T03:15:24.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 1 -- 172.21.15.153:0/1550645332 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3120137623 0 0) 0x7ff550071fa0 con 0x7ff55006fb00 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 1 -- 172.21.15.153:0/1550645332 >> v1:172.21.15.196:6789/0 conn(0x7ff5500af990 legacy=0x7ff55014b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 1 -- 172.21.15.153:0/1550645332 >> v1:172.21.15.153:6789/0 conn(0x7ff5500b0420 legacy=0x7ff550140500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 10 monclient: _finish_hunting 0 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 1 monclient: found mon.noname-b 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 1 -- 172.21.15.153:0/1550645332 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5500710a0 con 0x7ff55006fb00 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 10 monclient: _finish_auth 0 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 10 monclient: _check_auth_tickets 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.655+0000 7ff54effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.656+0000 7ff54effd640 1 -- 172.21.15.153:0/1550645332 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff53c001670 con 0x7ff55006fb00 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.656+0000 7ff54effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.656+0000 7ff54effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:15:24.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.656+0000 7ff54effd640 10 monclient: dump: 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.656+0000 7ff54effd640 1 -- 172.21.15.153:0/1550645332 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff53c0018c0 con 0x7ff55006fb00 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.656+0000 7ff54effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.656+0000 7ff54effd640 1 -- 172.21.15.153:0/1550645332 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff53c001c50 con 0x7ff55006fb00 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.656+0000 7ff54effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:24.657 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.656+0000 7ff54effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.656+0000 7ff54effd640 10 monclient: dump: 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.657+0000 7ff555201640 5 monclient: authenticate success, global_id 4634 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.657+0000 7ff555201640 10 monclient: get_monmap_and_config success 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.657+0000 7ff555201640 4 set_mon_vals no callback set 2024-03-27T03:15:24.658 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.658+0000 7ff555201640 10 monclient: shutdown 2024-03-27T03:15:24.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.658+0000 7ff555201640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:24.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.658+0000 7ff555201640 1 -- 172.21.15.153:0/1550645332 >> v1:172.21.15.153:6790/0 conn(0x7ff55006fb00 legacy=0x7ff55006fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:24.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.659+0000 7ff555201640 1 -- 172.21.15.153:0/1550645332 shutdown_connections 2024-03-27T03:15:24.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.659+0000 7ff555201640 1 -- 172.21.15.153:0/1550645332 >> 172.21.15.153:0/1550645332 conn(0x7ff5500672e0 msgr2=0x7ff5500b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:24.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.660+0000 7ff555201640 1 -- 172.21.15.153:0/1550645332 shutdown_connections 2024-03-27T03:15:24.659 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.660+0000 7ff555201640 1 -- 172.21.15.153:0/1550645332 wait complete. 2024-03-27T03:15:24.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.661+0000 7ff555201640 10 monclient: build_initial_monmap 2024-03-27T03:15:24.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.661+0000 7ff555201640 10 monclient: monmap: 2024-03-27T03:15:24.660 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:24.660 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:15:24.660 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:15:24.660 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:15:24.660 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:24.660 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:24.660 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:24.660 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:15:24.660 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:15:24.661 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:24.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.661+0000 7ff555201640 1 librados: starting msgr at 2024-03-27T03:15:24.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.661+0000 7ff555201640 1 librados: starting objecter 2024-03-27T03:15:24.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.661+0000 7ff555201640 1 Processor -- start 2024-03-27T03:15:24.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.664+0000 7ff555201640 1 -- start start 2024-03-27T03:15:24.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.664+0000 7ff555201640 1 librados: setting wanted keys 2024-03-27T03:15:24.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.664+0000 7ff555201640 1 librados: calling monclient init 2024-03-27T03:15:24.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.664+0000 7ff555201640 10 monclient: init 2024-03-27T03:15:24.662 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff555201640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff555201640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff555201640 10 monclient(hunting): picked mon.noname-b con 0x7ff55006fb00 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff555201640 10 monclient(hunting): picked mon.noname-c con 0x7ff5500af990 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff555201640 10 monclient(hunting): picked mon.noname-a con 0x7ff5500b0420 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff555201640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55013be80 con 0x7ff5500b0420 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff555201640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55013d0e0 con 0x7ff55006fb00 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff555201640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55013dec0 con 0x7ff5500af990 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff555201640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff555201640 10 monclient(hunting): authenticate will time out at 1884.973041s 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff54f7fe640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7ff5500af990 0x7ff5501dbbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:53550/0 (socket says 172.21.15.153:53550) 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff54f7fe640 1 -- 172.21.15.153:0/4167859517 learned_addr learned my addr 172.21.15.153:0/4167859517 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.666+0000 7ff54cff9640 10 client.?.objecter ms_handle_connect 0x7ff5500af990 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.667+0000 7ff54cff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.667+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985124602 0 0) 0x7ff55013dec0 con 0x7ff5500af990 2024-03-27T03:15:24.663 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.667+0000 7ff54cff9640 10 monclient(hunting): _negotiate 2024-03-27T03:15:24.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.667+0000 7ff54cff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:24.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.667+0000 7ff54cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:24.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.667+0000 7ff54cff9640 10 monclient(hunting): my global_id is 4637 2024-03-27T03:15:24.664 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.667+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff538004eb0 con 0x7ff5500af990 2024-03-27T03:15:24.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.669+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 382811417 0 0) 0x7ff538004eb0 con 0x7ff5500af990 2024-03-27T03:15:24.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.669+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff55013dec0 con 0x7ff5500af990 2024-03-27T03:15:24.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.669+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff540003fe0 con 0x7ff5500af990 2024-03-27T03:15:24.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.669+0000 7ff54cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:24.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.669+0000 7ff54cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:15:24.667 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.669+0000 7ff54cff9640 10 monclient(hunting): dump: 2024-03-27T03:15:24.667 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:24.667 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:24.667 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:24.667 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:24.667 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:24.667 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.669+0000 7ff54cff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.669+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3980416057 0 0) 0x7ff55013dec0 con 0x7ff5500af990 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.669+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 >> v1:172.21.15.196:6789/0 conn(0x7ff55006fb00 legacy=0x7ff55013b770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.671+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 >> v1:172.21.15.153:6789/0 conn(0x7ff5500b0420 legacy=0x7ff550139350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.671+0000 7ff54cff9640 10 monclient: _finish_hunting 0 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.671+0000 7ff54cff9640 1 monclient: found mon.c 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.671+0000 7ff54cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.671+0000 7ff54cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.671+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55013e050 con 0x7ff5500af990 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.672+0000 7ff54cff9640 10 monclient: _finish_auth 0 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.672+0000 7ff54cff9640 10 monclient: _check_auth_tickets 2024-03-27T03:15:24.668 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.672+0000 7ff54cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:24.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.672+0000 7ff555201640 5 monclient: authenticate success, global_id 4637 2024-03-27T03:15:24.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.672+0000 7ff555201640 10 monclient: _renew_subs 2024-03-27T03:15:24.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.672+0000 7ff555201640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:15:24.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.672+0000 7ff555201640 1 -- 172.21.15.153:0/4167859517 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5501dc2c0 con 0x7ff5500af990 2024-03-27T03:15:24.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.672+0000 7ff555201640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:15:24.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.672+0000 7ff555201640 10 monclient: _renew_subs 2024-03-27T03:15:24.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.672+0000 7ff555201640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:15:24.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.672+0000 7ff555201640 1 -- 172.21.15.153:0/4167859517 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff550139c40 con 0x7ff5500af990 2024-03-27T03:15:24.669 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.672+0000 7ff555201640 1 librados: init done 2024-03-27T03:15:24.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.674+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff540004140 con 0x7ff5500af990 2024-03-27T03:15:24.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.674+0000 7ff54cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:24.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.674+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7ff540001180 con 0x7ff5500af990 2024-03-27T03:15:24.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.674+0000 7ff54e7fc640 4 set_mon_vals no callback set 2024-03-27T03:15:24.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.674+0000 7ff555201640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:24.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.674+0000 7ff54cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:24.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.674+0000 7ff54cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:15:24.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.674+0000 7ff54cff9640 10 monclient: dump: 2024-03-27T03:15:24.670 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:24.670 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:24.671 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:24.671 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:24.671 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:24.671 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:24.671 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:24.671 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:24.671 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:24.671 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:24.671 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.676+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7ff5400012e0 con 0x7ff5500af990 2024-03-27T03:15:24.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.676+0000 7ff555201640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:24.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.676+0000 7ff555201640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:15:24.672 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.676+0000 7ff555201640 1 -- 172.21.15.153:0/4167859517 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52c005080 con 0x7ff5500af990 2024-03-27T03:15:24.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.677+0000 7ff54ffff640 10 monclient: get_auth_request con 0x7ff53806dc70 auth_method 0 2024-03-27T03:15:24.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.677+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7ff540084a90 con 0x7ff5500af990 2024-03-27T03:15:24.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.678+0000 7ff54cff9640 10 client.4637.objecter ms_dispatch 0x7ff550000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:15:24.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.678+0000 7ff54cff9640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:15:24.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.678+0000 7ff54cff9640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:15:24.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.678+0000 7ff54cff9640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-27T03:15:24.673 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.678+0000 7ff54cff9640 10 client.4637.objecter ms_handle_connect 0x7ff53806dc70 2024-03-27T03:15:24.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.686+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff540050370 con 0x7ff5500af990 2024-03-27T03:15:24.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.686+0000 7ff54cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:24.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:24.686+0000 7ff54cff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:15:25.005 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.012+0000 7ff555201640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:25.006 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.012+0000 7ff555201640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:25.006 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.012+0000 7ff555201640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:15:25.006 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.012+0000 7ff555201640 1 -- 172.21.15.153:0/4167859517 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff52c005330 con 0x7ff5500af990 2024-03-27T03:15:25.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.013+0000 7ff54cff9640 1 -- 172.21.15.153:0/4167859517 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7ff5400522a0 con 0x7ff5500af990 2024-03-27T03:15:25.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.013+0000 7ff54cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:25.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.013+0000 7ff54cff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:15:25.007 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:15:25.007 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:15:25.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.017+0000 7ff5467fc640 10 librados: watch_flush enter 2024-03-27T03:15:25.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.018+0000 7ff5467fc640 10 librados: watch_flush exit 2024-03-27T03:15:25.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.018+0000 7ff5467fc640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-27T03:15:25.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.018+0000 7ff5467fc640 10 client.4637.objecter successfully canceled tick 2024-03-27T03:15:25.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.018+0000 7ff5467fc640 1 -- 172.21.15.153:0/4167859517 >> v1:172.21.15.153:6816/2637937781 conn(0x7ff53806dc70 legacy=0x7ff538070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:25.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.018+0000 7ff5467fc640 10 monclient: shutdown 2024-03-27T03:15:25.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.018+0000 7ff5467fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:25.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.018+0000 7ff5467fc640 1 -- 172.21.15.153:0/4167859517 >> v1:172.21.15.153:6790/0 conn(0x7ff5500af990 legacy=0x7ff5501dbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:25.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.018+0000 7ff5467fc640 1 -- 172.21.15.153:0/4167859517 shutdown_connections 2024-03-27T03:15:25.011 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.018+0000 7ff5467fc640 1 -- 172.21.15.153:0/4167859517 >> 172.21.15.153:0/4167859517 conn(0x7ff5500672e0 msgr2=0x7ff550133040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:25.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.018+0000 7ff5467fc640 1 -- 172.21.15.153:0/4167859517 shutdown_connections 2024-03-27T03:15:25.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.018+0000 7ff5467fc640 1 -- 172.21.15.153:0/4167859517 wait complete. 2024-03-27T03:15:25.012 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:25.018+0000 7ff5467fc640 1 librados: shutdown 2024-03-27T03:15:25.034 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:15:31.842 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:15:31.970 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.977+0000 7f204a281640 10 monclient: get_monmap_and_config 2024-03-27T03:15:31.970 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.977+0000 7f204a281640 10 monclient: build_initial_monmap 2024-03-27T03:15:31.970 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.977+0000 7f204a281640 10 monclient: monmap: 2024-03-27T03:15:31.970 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:31.971 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:31.971 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:15:31.978184+0000 2024-03-27T03:15:31.971 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:15:31.978184+0000 2024-03-27T03:15:31.971 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:31.971 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:31.971 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:31.971 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:15:31.971 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:15:31.971 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 1 Processor -- start 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 1 -- start start 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 10 monclient: init 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 10 monclient(hunting): picked mon.noname-c con 0x7f20441457f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 10 monclient(hunting): picked mon.noname-a con 0x7f204413f650 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 10 monclient(hunting): picked mon.noname-b con 0x7f20440a9d70 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20440aaef0 con 0x7f204413f650 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2044145bd0 con 0x7f20441457f0 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2044145eb0 con 0x7f20440a9d70 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f204a281640 10 monclient(hunting): authenticate will time out at 1892.284568s 2024-03-27T03:15:31.973 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.978+0000 7f2043fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f20441457f0 0x7f204413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:43320/0 (socket says 172.21.15.153:43320) 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2043fff640 1 -- 172.21.15.153:0/3091306817 learned_addr learned my addr 172.21.15.153:0/3091306817 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2042ffd640 1 -- 172.21.15.153:0/3091306817 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151778133 0 0) 0x7f2044145eb0 con 0x7f20440a9d70 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2042ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2042ffd640 10 monclient(hunting): my global_id is 4640 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2042ffd640 1 -- 172.21.15.153:0/3091306817 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f202c003520 con 0x7f20440a9d70 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2042ffd640 1 -- 172.21.15.153:0/3091306817 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828046744 0 0) 0x7f2044145bd0 con 0x7f20441457f0 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2042ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2042ffd640 10 monclient(hunting): my global_id is 4582 2024-03-27T03:15:31.974 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.979+0000 7f2042ffd640 1 -- 172.21.15.153:0/3091306817 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2044145eb0 con 0x7f20441457f0 2024-03-27T03:15:31.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.980+0000 7f2042ffd640 1 -- 172.21.15.153:0/3091306817 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 910581467 0 0) 0x7f202c003520 con 0x7f20440a9d70 2024-03-27T03:15:31.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.980+0000 7f2042ffd640 1 -- 172.21.15.153:0/3091306817 >> v1:172.21.15.196:6789/0 conn(0x7f20441457f0 legacy=0x7f204413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:31.975 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.980+0000 7f2042ffd640 1 -- 172.21.15.153:0/3091306817 >> v1:172.21.15.153:6789/0 conn(0x7f204413f650 legacy=0x7f20440b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:15:31.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.982+0000 7f2042ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:15:31.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 1 monclient: found mon.noname-b 2024-03-27T03:15:31.976 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 1 -- 172.21.15.153:0/3091306817 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2044146340 con 0x7f20440a9d70 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 10 monclient: _finish_auth 0 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 1 -- 172.21.15.153:0/3091306817 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f203c001a10 con 0x7f20440a9d70 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 10 monclient: dump: 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:31.977 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f204a281640 5 monclient: authenticate success, global_id 4640 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f204a281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 1 -- 172.21.15.153:0/3091306817 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f203c001c30 con 0x7f20440a9d70 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 1 -- 172.21.15.153:0/3091306817 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f203c001fc0 con 0x7f20440a9d70 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f2042ffd640 10 monclient: dump: 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:31.978 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:31.979 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:31.979 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:31.979 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:31.979 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:31.979 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:31.979 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:31.979 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:31.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f204a281640 10 monclient: get_monmap_and_config success 2024-03-27T03:15:31.979 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.983+0000 7f204a281640 4 set_mon_vals no callback set 2024-03-27T03:15:31.985 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.984+0000 7f204a281640 10 monclient: shutdown 2024-03-27T03:15:31.985 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.984+0000 7f204a281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:31.985 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.984+0000 7f204a281640 1 -- 172.21.15.153:0/3091306817 >> v1:172.21.15.153:6790/0 conn(0x7f20440a9d70 legacy=0x7f204414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:31.985 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.985+0000 7f204a281640 1 -- 172.21.15.153:0/3091306817 shutdown_connections 2024-03-27T03:15:31.985 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.985+0000 7f204a281640 1 -- 172.21.15.153:0/3091306817 >> 172.21.15.153:0/3091306817 conn(0x7f2044066a00 msgr2=0x7f2044066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.985+0000 7f204a281640 1 -- 172.21.15.153:0/3091306817 shutdown_connections 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.985+0000 7f204a281640 1 -- 172.21.15.153:0/3091306817 wait complete. 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.986+0000 7f204a281640 10 monclient: build_initial_monmap 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.986+0000 7f204a281640 10 monclient: monmap: 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:15:31.986 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.987+0000 7f204a281640 1 librados: starting msgr at 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.987+0000 7f204a281640 1 librados: starting objecter 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.987+0000 7f204a281640 1 Processor -- start 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.987+0000 7f204a281640 1 -- start start 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.987+0000 7f204a281640 1 librados: setting wanted keys 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.987+0000 7f204a281640 1 librados: calling monclient init 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.987+0000 7f204a281640 10 monclient: init 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f204a281640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f204a281640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f204a281640 10 monclient(hunting): picked mon.noname-a con 0x7f20440a9d70 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f204a281640 10 monclient(hunting): picked mon.noname-b con 0x7f204413f650 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f204a281640 10 monclient(hunting): picked mon.noname-c con 0x7f20441457f0 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f204a281640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204406ec80 con 0x7f20440a9d70 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f204a281640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20441dfe10 con 0x7f204413f650 2024-03-27T03:15:31.987 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f204881e640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f20441457f0 0x7f20441df700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:58136/0 (socket says 172.21.15.153:58136) 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f204881e640 1 -- 172.21.15.153:0/4143259796 learned_addr learned my addr 172.21.15.153:0/4143259796 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f204a281640 1 -- 172.21.15.153:0/4143259796 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204406eef0 con 0x7f20441457f0 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f204a281640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f2040ff9640 10 client.?.objecter ms_handle_connect 0x7f20441457f0 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f2040ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f204a281640 10 monclient(hunting): authenticate will time out at 1892.294459s 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.988+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162790885 0 0) 0x7f204406eef0 con 0x7f20441457f0 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 10 monclient(hunting): _negotiate 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 10 monclient(hunting): my global_id is 4643 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2030004eb0 con 0x7f20441457f0 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 301514334 0 0) 0x7f2030004eb0 con 0x7f20441457f0 2024-03-27T03:15:31.988 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f204406eef0 con 0x7f20441457f0 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f203c001a10 con 0x7f20441457f0 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 10 monclient(hunting): dump: 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:15:31.989 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2684850148 0 0) 0x7f204406eef0 con 0x7f20441457f0 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 >> v1:172.21.15.196:6789/0 conn(0x7f204413f650 legacy=0x7f20441deff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.989+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 >> v1:172.21.15.153:6789/0 conn(0x7f20440a9d70 legacy=0x7f2044141b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.990+0000 7f2040ff9640 10 monclient: _finish_hunting 0 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.990+0000 7f2040ff9640 1 monclient: found mon.c 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.990+0000 7f2040ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.990+0000 7f2040ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.991+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204406f1d0 con 0x7f20441457f0 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.991+0000 7f2040ff9640 10 monclient: _finish_auth 0 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.991+0000 7f2040ff9640 10 monclient: _check_auth_tickets 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.991+0000 7f2040ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.991+0000 7f204a281640 5 monclient: authenticate success, global_id 4643 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.991+0000 7f204a281640 10 monclient: _renew_subs 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.991+0000 7f204a281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.991+0000 7f204a281640 1 -- 172.21.15.153:0/4143259796 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204406e0a0 con 0x7f20441457f0 2024-03-27T03:15:31.990 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.991+0000 7f204a281640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.991+0000 7f204a281640 10 monclient: _renew_subs 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.991+0000 7f204a281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.992+0000 7f204a281640 1 -- 172.21.15.153:0/4143259796 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2044137bf0 con 0x7f20441457f0 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.992+0000 7f204a281640 1 librados: init done 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.992+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f203c003c20 con 0x7f20441457f0 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.992+0000 7f2040ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.992+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f203c004070 con 0x7f20441457f0 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.992+0000 7f2040ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.992+0000 7f2040ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.992+0000 7f2040ff9640 10 monclient: dump: 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:31.991 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:31.992 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:31.992 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:31.992 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:31.992 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:31.992 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.992+0000 7f2041ffb640 4 set_mon_vals no callback set 2024-03-27T03:15:31.993 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.993+0000 7f204a281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:31.993 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.993+0000 7f204a281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:31.993 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.993+0000 7f204a281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:15:31.993 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.993+0000 7f204a281640 1 -- 172.21.15.153:0/4143259796 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2024005080 con 0x7f20441457f0 2024-03-27T03:15:31.993 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.994+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f203c0041d0 con 0x7f20441457f0 2024-03-27T03:15:31.993 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.995+0000 7f2043fff640 10 monclient: get_auth_request con 0x7f203006dbe0 auth_method 0 2024-03-27T03:15:31.993 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.995+0000 7f2040ff9640 10 client.4643.objecter ms_handle_connect 0x7f203006dbe0 2024-03-27T03:15:31.993 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.995+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 602960344 0 0) 0x7f203c084f70 con 0x7f20441457f0 2024-03-27T03:15:31.993 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.995+0000 7f2040ff9640 10 client.4643.objecter ms_dispatch 0x7f2044000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T03:15:31.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.995+0000 7f2040ff9640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T03:15:31.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.995+0000 7f2040ff9640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-27T03:15:31.994 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:31.995+0000 7f2040ff9640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-27T03:15:31.995 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.002+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f203c050850 con 0x7f20441457f0 2024-03-27T03:15:31.995 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.002+0000 7f2040ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:31.995 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.002+0000 7f2040ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:15:32.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.310+0000 7f204a281640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:32.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.310+0000 7f204a281640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:32.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.310+0000 7f204a281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:15:32.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.310+0000 7f204a281640 1 -- 172.21.15.153:0/4143259796 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2024005330 con 0x7f20441457f0 2024-03-27T03:15:32.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.312+0000 7f2040ff9640 1 -- 172.21.15.153:0/4143259796 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2431861901) 0x7f203c052780 con 0x7f20441457f0 2024-03-27T03:15:32.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.312+0000 7f2040ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:32.305 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.312+0000 7f2040ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:15:32.305 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:15:32.306 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:05:34.717637+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:05:34.717637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:15:32.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.315+0000 7f204a281640 10 librados: watch_flush enter 2024-03-27T03:15:32.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.315+0000 7f204a281640 10 librados: watch_flush exit 2024-03-27T03:15:32.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.315+0000 7f204a281640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-27T03:15:32.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.315+0000 7f204a281640 10 client.4643.objecter successfully canceled tick 2024-03-27T03:15:32.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.315+0000 7f204a281640 1 -- 172.21.15.153:0/4143259796 >> v1:172.21.15.153:6816/2637937781 conn(0x7f203006dbe0 legacy=0x7f20300700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:32.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.315+0000 7f204a281640 10 monclient: shutdown 2024-03-27T03:15:32.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.315+0000 7f204a281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:32.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.315+0000 7f204a281640 1 -- 172.21.15.153:0/4143259796 >> v1:172.21.15.153:6790/0 conn(0x7f20441457f0 legacy=0x7f20441df700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:32.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.315+0000 7f204a281640 1 -- 172.21.15.153:0/4143259796 shutdown_connections 2024-03-27T03:15:32.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.315+0000 7f204a281640 1 -- 172.21.15.153:0/4143259796 >> 172.21.15.153:0/4143259796 conn(0x7f2044066a00 msgr2=0x7f20440b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:32.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.315+0000 7f204a281640 1 -- 172.21.15.153:0/4143259796 shutdown_connections 2024-03-27T03:15:32.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.315+0000 7f204a281640 1 -- 172.21.15.153:0/4143259796 wait complete. 2024-03-27T03:15:32.309 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:32.316+0000 7f204a281640 1 librados: shutdown 2024-03-27T03:15:32.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:15:39.130 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:15:39.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.281+0000 7fd75ab40640 10 monclient: get_monmap_and_config 2024-03-27T03:15:39.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.281+0000 7fd75ab40640 10 monclient: build_initial_monmap 2024-03-27T03:15:39.275 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.281+0000 7fd75ab40640 10 monclient: monmap: 2024-03-27T03:15:39.276 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:39.276 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:39.276 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:15:39.282293+0000 2024-03-27T03:15:39.276 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:15:39.282293+0000 2024-03-27T03:15:39.276 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:39.276 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:39.276 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:39.276 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:15:39.276 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:15:39.276 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.282+0000 7fd75ab40640 1 Processor -- start 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.283+0000 7fd75ab40640 1 -- start start 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.283+0000 7fd75ab40640 10 monclient: init 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.284+0000 7fd75ab40640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.284+0000 7fd75ab40640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.284+0000 7fd75ab40640 10 monclient(hunting): picked mon.noname-a con 0x7fd7540a9930 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.284+0000 7fd75ab40640 10 monclient(hunting): picked mon.noname-c con 0x7fd754149dc0 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.284+0000 7fd75ab40640 10 monclient(hunting): picked mon.noname-b con 0x7fd75414ff40 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.284+0000 7fd75ab40640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7540aae60 con 0x7fd7540a9930 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.284+0000 7fd75ab40640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7540b4310 con 0x7fd754149dc0 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.284+0000 7fd75ab40640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7540afdd0 con 0x7fd75414ff40 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.284+0000 7fd75ab40640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:39.278 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.284+0000 7fd75ab40640 10 monclient(hunting): authenticate will time out at 1899.590040s 2024-03-27T03:15:39.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.287+0000 7fd7588dc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd7540a9930 0x7fd754143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:55247/0 (socket says 172.21.15.153:55247) 2024-03-27T03:15:39.280 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.287+0000 7fd7588dc640 1 -- 172.21.15.153:0/3216907052 learned_addr learned my addr 172.21.15.153:0/3216907052 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:39.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.287+0000 7fd7537fe640 1 -- 172.21.15.153:0/3216907052 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216797129 0 0) 0x7fd7540b4310 con 0x7fd754149dc0 2024-03-27T03:15:39.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.287+0000 7fd7537fe640 10 monclient(hunting): _negotiate 2024-03-27T03:15:39.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.287+0000 7fd7537fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:39.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.287+0000 7fd7537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:39.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.287+0000 7fd7537fe640 10 monclient(hunting): my global_id is 4585 2024-03-27T03:15:39.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.287+0000 7fd7537fe640 1 -- 172.21.15.153:0/3216907052 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd740003520 con 0x7fd754149dc0 2024-03-27T03:15:39.283 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 1 -- 172.21.15.153:0/3216907052 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075248967 0 0) 0x7fd7540aae60 con 0x7fd7540a9930 2024-03-27T03:15:39.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient(hunting): _negotiate 2024-03-27T03:15:39.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:39.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:39.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient(hunting): my global_id is 4551 2024-03-27T03:15:39.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 1 -- 172.21.15.153:0/3216907052 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7540b4310 con 0x7fd7540a9930 2024-03-27T03:15:39.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 1 -- 172.21.15.153:0/3216907052 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 939931748 0 0) 0x7fd7540b4310 con 0x7fd7540a9930 2024-03-27T03:15:39.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 1 -- 172.21.15.153:0/3216907052 >> v1:172.21.15.153:6790/0 conn(0x7fd75414ff40 legacy=0x7fd754152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:39.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 1 -- 172.21.15.153:0/3216907052 >> v1:172.21.15.196:6789/0 conn(0x7fd754149dc0 legacy=0x7fd75414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:39.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient: _finish_hunting 0 2024-03-27T03:15:39.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 1 monclient: found mon.noname-a 2024-03-27T03:15:39.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:39.284 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 1 -- 172.21.15.153:0/3216907052 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75414a950 con 0x7fd7540a9930 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient: _finish_auth 0 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient: _check_auth_tickets 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 1 -- 172.21.15.153:0/3216907052 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd73c001670 con 0x7fd7540a9930 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient: dump: 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:39.285 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd75ab40640 5 monclient: authenticate success, global_id 4551 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd75ab40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 1 -- 172.21.15.153:0/3216907052 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd73c0017d0 con 0x7fd7540a9930 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 1 -- 172.21.15.153:0/3216907052 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd73c001b60 con 0x7fd7540a9930 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.288+0000 7fd7537fe640 10 monclient: dump: 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:39.286 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.289+0000 7fd75ab40640 10 monclient: get_monmap_and_config success 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.289+0000 7fd75ab40640 4 set_mon_vals no callback set 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.289+0000 7fd75ab40640 10 monclient: shutdown 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.289+0000 7fd75ab40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.289+0000 7fd75ab40640 1 -- 172.21.15.153:0/3216907052 >> v1:172.21.15.153:6789/0 conn(0x7fd7540a9930 legacy=0x7fd754143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.290+0000 7fd75ab40640 1 -- 172.21.15.153:0/3216907052 shutdown_connections 2024-03-27T03:15:39.287 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.290+0000 7fd75ab40640 1 -- 172.21.15.153:0/3216907052 >> 172.21.15.153:0/3216907052 conn(0x7fd7540b35e0 msgr2=0x7fd7540b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:39.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.292+0000 7fd75ab40640 1 -- 172.21.15.153:0/3216907052 shutdown_connections 2024-03-27T03:15:39.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.292+0000 7fd75ab40640 1 -- 172.21.15.153:0/3216907052 wait complete. 2024-03-27T03:15:39.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.293+0000 7fd75ab40640 10 monclient: build_initial_monmap 2024-03-27T03:15:39.289 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.293+0000 7fd75ab40640 10 monclient: monmap: 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.294+0000 7fd75ab40640 1 librados: starting msgr at 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.294+0000 7fd75ab40640 1 librados: starting objecter 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.294+0000 7fd75ab40640 1 Processor -- start 2024-03-27T03:15:39.290 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.294+0000 7fd75ab40640 1 -- start start 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.294+0000 7fd75ab40640 1 librados: setting wanted keys 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.294+0000 7fd75ab40640 1 librados: calling monclient init 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.294+0000 7fd75ab40640 10 monclient: init 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.295+0000 7fd75ab40640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.295+0000 7fd75ab40640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.295+0000 7fd75ab40640 10 monclient(hunting): picked mon.noname-c con 0x7fd754149dc0 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.295+0000 7fd75ab40640 10 monclient(hunting): picked mon.noname-b con 0x7fd75414ff40 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.295+0000 7fd75ab40640 10 monclient(hunting): picked mon.noname-a con 0x7fd75406fef0 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.295+0000 7fd75ab40640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75406e810 con 0x7fd75406fef0 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.295+0000 7fd75ab40640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd754071bd0 con 0x7fd75414ff40 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.295+0000 7fd75ab40640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75413f680 con 0x7fd754149dc0 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.295+0000 7fd75ab40640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:39.291 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.295+0000 7fd75ab40640 10 monclient(hunting): authenticate will time out at 1899.601093s 2024-03-27T03:15:39.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.298+0000 7fd753fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd75414ff40 0x7fd75406f7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:48808/0 (socket says 172.21.15.153:48808) 2024-03-27T03:15:39.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.298+0000 7fd753fff640 1 -- 172.21.15.153:0/351875249 learned_addr learned my addr 172.21.15.153:0/351875249 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:39.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 client.?.objecter ms_handle_connect 0x7fd75406fef0 2024-03-27T03:15:39.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:39.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 client.?.objecter ms_handle_connect 0x7fd75414ff40 2024-03-27T03:15:39.292 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:39.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 client.?.objecter ms_handle_connect 0x7fd754149dc0 2024-03-27T03:15:39.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:39.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226019505 0 0) 0x7fd754071bd0 con 0x7fd75414ff40 2024-03-27T03:15:39.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 monclient(hunting): _negotiate 2024-03-27T03:15:39.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:39.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:39.296 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 monclient(hunting): my global_id is 4588 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd734004eb0 con 0x7fd75414ff40 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413736038 0 0) 0x7fd75406e810 con 0x7fd75406fef0 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 monclient(hunting): _negotiate 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 10 monclient(hunting): my global_id is 4554 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.299+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd754071bd0 con 0x7fd75406fef0 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.300+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992749617 0 0) 0x7fd75413f680 con 0x7fd754149dc0 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.300+0000 7fd7517fa640 10 monclient(hunting): _negotiate 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.300+0000 7fd7517fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.300+0000 7fd7517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.300+0000 7fd7517fa640 10 monclient(hunting): my global_id is 4649 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.300+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd75406e810 con 0x7fd754149dc0 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.300+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1691955389 0 0) 0x7fd754071bd0 con 0x7fd75406fef0 2024-03-27T03:15:39.297 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.300+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd75413f680 con 0x7fd75406fef0 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.300+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd74c001b10 con 0x7fd75406fef0 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.301+0000 7fd7517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.301+0000 7fd7517fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.301+0000 7fd7517fa640 10 monclient(hunting): dump: 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:39.298 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.301+0000 7fd7517fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.301+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4279777739 0 0) 0x7fd75406e810 con 0x7fd754149dc0 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.301+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd754071bd0 con 0x7fd754149dc0 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.301+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 97146996 0 0) 0x7fd75413f680 con 0x7fd75406fef0 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.301+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 >> v1:172.21.15.153:6790/0 conn(0x7fd754149dc0 legacy=0x7fd7540720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.301+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 >> v1:172.21.15.196:6789/0 conn(0x7fd75414ff40 legacy=0x7fd75406f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.301+0000 7fd7517fa640 10 monclient: _finish_hunting 0 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.301+0000 7fd7517fa640 1 monclient: found mon.a 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.302+0000 7fd7517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.302+0000 7fd7517fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.302+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75406eed0 con 0x7fd75406fef0 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.302+0000 7fd7517fa640 10 monclient: _finish_auth 0 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.302+0000 7fd7517fa640 10 monclient: _check_auth_tickets 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.302+0000 7fd7517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.302+0000 7fd75ab40640 5 monclient: authenticate success, global_id 4554 2024-03-27T03:15:39.299 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.302+0000 7fd75ab40640 10 monclient: _renew_subs 2024-03-27T03:15:39.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.302+0000 7fd75ab40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:39.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.302+0000 7fd75ab40640 1 -- 172.21.15.153:0/351875249 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7541e76d0 con 0x7fd75406fef0 2024-03-27T03:15:39.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.303+0000 7fd75ab40640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:15:39.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.303+0000 7fd75ab40640 10 monclient: _renew_subs 2024-03-27T03:15:39.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.303+0000 7fd75ab40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:39.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.303+0000 7fd75ab40640 1 -- 172.21.15.153:0/351875249 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7541e7a40 con 0x7fd75406fef0 2024-03-27T03:15:39.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.303+0000 7fd75ab40640 1 librados: init done 2024-03-27T03:15:39.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.303+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd74c001f80 con 0x7fd75406fef0 2024-03-27T03:15:39.300 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.303+0000 7fd7517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:39.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.303+0000 7fd752ffd640 4 set_mon_vals no callback set 2024-03-27T03:15:39.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.304+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fd74c003b10 con 0x7fd75406fef0 2024-03-27T03:15:39.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.304+0000 7fd7517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:39.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.304+0000 7fd7517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:15:39.302 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.304+0000 7fd7517fa640 10 monclient: dump: 2024-03-27T03:15:39.302 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:39.302 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:39.302 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:39.302 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:39.302 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.305+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fd74c003d30 con 0x7fd75406fef0 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.306+0000 7fd75ab40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.306+0000 7fd75ab40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.306+0000 7fd75ab40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.306+0000 7fd75ab40640 1 -- 172.21.15.153:0/351875249 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd738005080 con 0x7fd75406fef0 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.306+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2380031179 0 0) 0x7fd74c0838f0 con 0x7fd75406fef0 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.306+0000 7fd7517fa640 10 client.4554.objecter ms_dispatch 0x7fd754000b90 osd_map(11..11 src has 1..11) v4 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.306+0000 7fd7517fa640 3 client.4554.objecter handle_osd_map got epochs [11,11] > 0 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.306+0000 7fd7517fa640 3 client.4554.objecter handle_osd_map decoding full epoch 11 2024-03-27T03:15:39.303 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.306+0000 7fd7517fa640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-27T03:15:39.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.306+0000 7fd7588dc640 10 monclient: get_auth_request con 0x7fd73406dfd0 auth_method 0 2024-03-27T03:15:39.304 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.307+0000 7fd7517fa640 10 client.4554.objecter ms_handle_connect 0x7fd73406dfd0 2024-03-27T03:15:39.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.315+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd74c04f1d0 con 0x7fd75406fef0 2024-03-27T03:15:39.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.315+0000 7fd7517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:39.308 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.315+0000 7fd7517fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:15:39.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.655+0000 7fd75ab40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:39.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.655+0000 7fd75ab40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:39.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.655+0000 7fd75ab40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:39.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.655+0000 7fd75ab40640 1 -- 172.21.15.153:0/351875249 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd738005330 con 0x7fd75406fef0 2024-03-27T03:15:39.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.656+0000 7fd7517fa640 1 -- 172.21.15.153:0/351875249 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 992601923) 0x7fd74c051100 con 0x7fd75406fef0 2024-03-27T03:15:39.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.657+0000 7fd7517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:39.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.657+0000 7fd7517fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:15:39.651 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:15:39.651 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":11,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:15:37.345901+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:15:37.345901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:15:39.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.660+0000 7fd75ab40640 10 librados: watch_flush enter 2024-03-27T03:15:39.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.661+0000 7fd75ab40640 10 librados: watch_flush exit 2024-03-27T03:15:39.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.661+0000 7fd75ab40640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-27T03:15:39.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.661+0000 7fd75ab40640 10 client.4554.objecter successfully canceled tick 2024-03-27T03:15:39.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.661+0000 7fd75ab40640 1 -- 172.21.15.153:0/351875249 >> v1:172.21.15.153:6816/2637937781 conn(0x7fd73406dfd0 legacy=0x7fd734070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:39.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.661+0000 7fd75ab40640 10 monclient: shutdown 2024-03-27T03:15:39.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.661+0000 7fd75ab40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:39.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.661+0000 7fd75ab40640 1 -- 172.21.15.153:0/351875249 >> v1:172.21.15.153:6789/0 conn(0x7fd75406fef0 legacy=0x7fd7541e5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:39.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.661+0000 7fd75ab40640 1 -- 172.21.15.153:0/351875249 shutdown_connections 2024-03-27T03:15:39.655 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.661+0000 7fd75ab40640 1 -- 172.21.15.153:0/351875249 >> 172.21.15.153:0/351875249 conn(0x7fd7540b35e0 msgr2=0x7fd7540b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:39.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.663+0000 7fd75ab40640 1 -- 172.21.15.153:0/351875249 shutdown_connections 2024-03-27T03:15:39.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.663+0000 7fd75ab40640 1 -- 172.21.15.153:0/351875249 wait complete. 2024-03-27T03:15:39.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:39.663+0000 7fd75ab40640 1 librados: shutdown 2024-03-27T03:15:39.673 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:15:46.480 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:15:46.634 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.640+0000 7fcacbc4a640 10 monclient: get_monmap_and_config 2024-03-27T03:15:46.634 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.640+0000 7fcacbc4a640 10 monclient: build_initial_monmap 2024-03-27T03:15:46.635 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.640+0000 7fcacbc4a640 10 monclient: monmap: 2024-03-27T03:15:46.635 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:46.635 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:46.635 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:15:46.641393+0000 2024-03-27T03:15:46.635 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:15:46.641393+0000 2024-03-27T03:15:46.635 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:46.635 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:46.635 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:46.635 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:15:46.635 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:15:46.635 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:46.636 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.642+0000 7fcacbc4a640 1 Processor -- start 2024-03-27T03:15:46.636 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.642+0000 7fcacbc4a640 1 -- start start 2024-03-27T03:15:46.636 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.642+0000 7fcacbc4a640 10 monclient: init 2024-03-27T03:15:46.636 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.643+0000 7fcacbc4a640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:46.636 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.643+0000 7fcacbc4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:15:46.637 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.643+0000 7fcacbc4a640 10 monclient(hunting): picked mon.noname-c con 0x7fcac40a9ac0 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:46.637 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.644+0000 7fcacbc4a640 10 monclient(hunting): picked mon.noname-b con 0x7fcac413ee90 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:46.640 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.647+0000 7fcacbc4a640 10 monclient(hunting): picked mon.noname-a con 0x7fcac40af900 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:46.640 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.647+0000 7fcaca1e7640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fcac40af900 0x7fcac4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:52545/0 (socket says 172.21.15.153:52545) 2024-03-27T03:15:46.641 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.647+0000 7fcaca1e7640 1 -- 172.21.15.153:0/3880863225 learned_addr learned my addr 172.21.15.153:0/3880863225 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:46.641 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.647+0000 7fcacbc4a640 1 -- 172.21.15.153:0/3880863225 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac40afd10 con 0x7fcac40af900 2024-03-27T03:15:46.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.648+0000 7fcacbc4a640 1 -- 172.21.15.153:0/3880863225 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac40af6f0 con 0x7fcac40a9ac0 2024-03-27T03:15:46.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.648+0000 7fcacbc4a640 1 -- 172.21.15.153:0/3880863225 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac40aac80 con 0x7fcac413ee90 2024-03-27T03:15:46.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.648+0000 7fcac89e4640 1 -- 172.21.15.153:0/3880863225 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820698848 0 0) 0x7fcac40afd10 con 0x7fcac40af900 2024-03-27T03:15:46.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.648+0000 7fcacbc4a640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:46.642 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.648+0000 7fcacbc4a640 10 monclient(hunting): authenticate will time out at 1906.953975s 2024-03-27T03:15:46.644 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.649+0000 7fcac89e4640 10 monclient(hunting): _negotiate 2024-03-27T03:15:46.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.649+0000 7fcac89e4640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:46.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.649+0000 7fcac89e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:46.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.649+0000 7fcac89e4640 10 monclient(hunting): my global_id is 4557 2024-03-27T03:15:46.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.649+0000 7fcac89e4640 1 -- 172.21.15.153:0/3880863225 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcab0003520 con 0x7fcac40af900 2024-03-27T03:15:46.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.651+0000 7fcac89e4640 1 -- 172.21.15.153:0/3880863225 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748135350 0 0) 0x7fcac40aac80 con 0x7fcac413ee90 2024-03-27T03:15:46.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.651+0000 7fcac89e4640 10 monclient(hunting): _negotiate 2024-03-27T03:15:46.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.651+0000 7fcac89e4640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:46.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.651+0000 7fcac89e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:46.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.651+0000 7fcac89e4640 10 monclient(hunting): my global_id is 4652 2024-03-27T03:15:46.645 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.651+0000 7fcac89e4640 1 -- 172.21.15.153:0/3880863225 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac40afd10 con 0x7fcac413ee90 2024-03-27T03:15:46.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 1 -- 172.21.15.153:0/3880863225 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1526877640 0 0) 0x7fcab0003520 con 0x7fcac40af900 2024-03-27T03:15:46.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 1 -- 172.21.15.153:0/3880863225 >> v1:172.21.15.153:6790/0 conn(0x7fcac413ee90 legacy=0x7fcac413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:46.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 1 -- 172.21.15.153:0/3880863225 >> v1:172.21.15.196:6789/0 conn(0x7fcac40a9ac0 legacy=0x7fcac40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:15:46.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 10 monclient: _finish_hunting 0 2024-03-27T03:15:46.646 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 1 monclient: found mon.noname-a 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 1 -- 172.21.15.153:0/3880863225 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac40aaf60 con 0x7fcac40af900 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 10 monclient: _finish_auth 0 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 10 monclient: _check_auth_tickets 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 1 -- 172.21.15.153:0/3880863225 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fcabc001a10 con 0x7fcac40af900 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 10 monclient: dump: 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:46.647 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:46.648 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:46.648 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:46.648 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:46.648 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:46.648 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:46.648 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:46.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 1 -- 172.21.15.153:0/3880863225 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcabc001c30 con 0x7fcac40af900 2024-03-27T03:15:46.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:46.648 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 1 -- 172.21.15.153:0/3880863225 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fcabc001fc0 con 0x7fcac40af900 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.652+0000 7fcac89e4640 10 monclient: dump: 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:46.649 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:46.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.656+0000 7fcacbc4a640 5 monclient: authenticate success, global_id 4557 2024-03-27T03:15:46.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.656+0000 7fcacbc4a640 10 monclient: get_monmap_and_config success 2024-03-27T03:15:46.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.656+0000 7fcacbc4a640 4 set_mon_vals no callback set 2024-03-27T03:15:46.650 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.657+0000 7fcacbc4a640 10 monclient: shutdown 2024-03-27T03:15:46.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.657+0000 7fcacbc4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:46.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.657+0000 7fcacbc4a640 1 -- 172.21.15.153:0/3880863225 >> v1:172.21.15.153:6789/0 conn(0x7fcac40af900 legacy=0x7fcac4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:46.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.657+0000 7fcacbc4a640 1 -- 172.21.15.153:0/3880863225 shutdown_connections 2024-03-27T03:15:46.651 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.657+0000 7fcacbc4a640 1 -- 172.21.15.153:0/3880863225 >> 172.21.15.153:0/3880863225 conn(0x7fcac40b35e0 msgr2=0x7fcac40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:46.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.661+0000 7fcacbc4a640 1 -- 172.21.15.153:0/3880863225 shutdown_connections 2024-03-27T03:15:46.654 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.661+0000 7fcacbc4a640 1 -- 172.21.15.153:0/3880863225 wait complete. 2024-03-27T03:15:46.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.662+0000 7fcacbc4a640 10 monclient: build_initial_monmap 2024-03-27T03:15:46.656 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.662+0000 7fcacbc4a640 10 monclient: monmap: 2024-03-27T03:15:46.656 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:46.656 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:15:46.656 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:15:46.656 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:15:46.656 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:46.657 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:46.657 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:46.657 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:15:46.657 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:15:46.657 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:46.657 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.662+0000 7fcacbc4a640 1 librados: starting msgr at 2024-03-27T03:15:46.657 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.662+0000 7fcacbc4a640 1 librados: starting objecter 2024-03-27T03:15:46.657 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.662+0000 7fcacbc4a640 1 Processor -- start 2024-03-27T03:15:46.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.663+0000 7fcacbc4a640 1 -- start start 2024-03-27T03:15:46.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.663+0000 7fcacbc4a640 1 librados: setting wanted keys 2024-03-27T03:15:46.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.663+0000 7fcacbc4a640 1 librados: calling monclient init 2024-03-27T03:15:46.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.663+0000 7fcacbc4a640 10 monclient: init 2024-03-27T03:15:46.660 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.664+0000 7fcacbc4a640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:46.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.664+0000 7fcacbc4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T03:15:46.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.664+0000 7fcacbc4a640 10 monclient(hunting): picked mon.noname-c con 0x7fcac40a9ac0 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:46.661 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.667+0000 7fcacbc4a640 10 monclient(hunting): picked mon.noname-a con 0x7fcac40af900 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:46.662 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.667+0000 7fcac91e5640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fcac40af900 0x7fcac413c7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:52981/0 (socket says 172.21.15.153:52981) 2024-03-27T03:15:46.662 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.668+0000 7fcac91e5640 1 -- 172.21.15.153:0/401366619 learned_addr learned my addr 172.21.15.153:0/401366619 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:46.670 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.669+0000 7fcacbc4a640 10 monclient(hunting): picked mon.noname-b con 0x7fcac413ee90 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.669+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac413dcb0 con 0x7fcac40af900 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.669+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac41421f0 con 0x7fcac413ee90 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.669+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac41424d0 con 0x7fcac40a9ac0 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.669+0000 7fcacbc4a640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.669+0000 7fcacbc4a640 10 monclient(hunting): authenticate will time out at 1906.974763s 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.671+0000 7fcaba7fc640 10 client.?.objecter ms_handle_connect 0x7fcac40af900 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.671+0000 7fcaba7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.671+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888771156 0 0) 0x7fcac413dcb0 con 0x7fcac40af900 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.671+0000 7fcaba7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.671+0000 7fcaba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.671+0000 7fcaba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.671+0000 7fcaba7fc640 10 monclient(hunting): my global_id is 4560 2024-03-27T03:15:46.674 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.671+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa8004eb0 con 0x7fcac40af900 2024-03-27T03:15:46.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.671+0000 7fcaba7fc640 10 client.?.objecter ms_handle_connect 0x7fcac40a9ac0 2024-03-27T03:15:46.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.671+0000 7fcaba7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:46.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093812610 0 0) 0x7fcac41424d0 con 0x7fcac40a9ac0 2024-03-27T03:15:46.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:15:46.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:46.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:46.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): my global_id is 4655 2024-03-27T03:15:46.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac413dcb0 con 0x7fcac40a9ac0 2024-03-27T03:15:46.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1871035326 0 0) 0x7fcac413dcb0 con 0x7fcac40a9ac0 2024-03-27T03:15:46.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcac41424d0 con 0x7fcac40a9ac0 2024-03-27T03:15:46.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 client.?.objecter ms_handle_connect 0x7fcac413ee90 2024-03-27T03:15:46.675 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fcac0001510 con 0x7fcac40a9ac0 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): dump: 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:46.676 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118089805 0 0) 0x7fcac41421f0 con 0x7fcac413ee90 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): _negotiate 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): my global_id is 4591 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac413dcb0 con 0x7fcac413ee90 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2828422873 0 0) 0x7fcaa8004eb0 con 0x7fcac40af900 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcac41421f0 con 0x7fcac40af900 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fcab40019a0 con 0x7fcac40af900 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.672+0000 7fcaba7fc640 10 monclient(hunting): dump: 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:46.677 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2326866678 0 0) 0x7fcac41421f0 con 0x7fcac40af900 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 >> v1:172.21.15.153:6790/0 conn(0x7fcac40a9ac0 legacy=0x7fcac4140e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 >> v1:172.21.15.196:6789/0 conn(0x7fcac413ee90 legacy=0x7fcac413cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcaba7fc640 10 monclient: _finish_hunting 0 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcaba7fc640 1 monclient: found mon.a 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcaba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:46.678 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcaba7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac41408b0 con 0x7fcac40af900 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcaba7fc640 10 monclient: _finish_auth 0 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcaba7fc640 10 monclient: _check_auth_tickets 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcaba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcacbc4a640 5 monclient: authenticate success, global_id 4560 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcacbc4a640 10 monclient: _renew_subs 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcacbc4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac4141570 con 0x7fcac40af900 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcacbc4a640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcacbc4a640 10 monclient: _renew_subs 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcacbc4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac41e7150 con 0x7fcac40af900 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.673+0000 7fcacbc4a640 1 librados: init done 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.674+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcab4003610 con 0x7fcac40af900 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.674+0000 7fcaba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:46.679 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.674+0000 7fcabbfff640 4 set_mon_vals no callback set 2024-03-27T03:15:46.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.675+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fcab4003a60 con 0x7fcac40af900 2024-03-27T03:15:46.680 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.676+0000 7fcaba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:46.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.676+0000 7fcaba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:15:46.681 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.676+0000 7fcaba7fc640 10 monclient: dump: 2024-03-27T03:15:46.681 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:46.681 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:46.681 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:46.681 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:46.682 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:46.682 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:46.682 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:46.682 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:46.682 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:46.682 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:46.682 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.676+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fcab4003c80 con 0x7fcac40af900 2024-03-27T03:15:46.682 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.677+0000 7fcacbc4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:46.683 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.677+0000 7fcacbc4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:46.683 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.677+0000 7fcacbc4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:46.683 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.677+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaac005080 con 0x7fcac40af900 2024-03-27T03:15:46.683 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.679+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2380031179 0 0) 0x7fcab4084a70 con 0x7fcac40af900 2024-03-27T03:15:46.683 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.679+0000 7fcaba7fc640 10 client.4560.objecter ms_dispatch 0x7fcac4000b90 osd_map(11..11 src has 1..11) v4 2024-03-27T03:15:46.683 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.679+0000 7fcaba7fc640 3 client.4560.objecter handle_osd_map got epochs [11,11] > 0 2024-03-27T03:15:46.683 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.679+0000 7fcaba7fc640 3 client.4560.objecter handle_osd_map decoding full epoch 11 2024-03-27T03:15:46.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.679+0000 7fcaba7fc640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-27T03:15:46.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.680+0000 7fcac99e6640 10 monclient: get_auth_request con 0x7fcaa806df70 auth_method 0 2024-03-27T03:15:46.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.682+0000 7fcaba7fc640 10 client.4560.objecter ms_handle_connect 0x7fcaa806df70 2024-03-27T03:15:46.684 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.689+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcab4050350 con 0x7fcac40af900 2024-03-27T03:15:46.685 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.689+0000 7fcaba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:46.685 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:46.689+0000 7fcaba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:15:47.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.015+0000 7fcacbc4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:47.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.015+0000 7fcacbc4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:47.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.015+0000 7fcacbc4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:47.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.015+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcaac005330 con 0x7fcac40af900 2024-03-27T03:15:47.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.017+0000 7fcaba7fc640 1 -- 172.21.15.153:0/401366619 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 992601923) 0x7fcab4052280 con 0x7fcac40af900 2024-03-27T03:15:47.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.017+0000 7fcaba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:47.010 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.017+0000 7fcaba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:15:47.010 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:15:47.010 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":11,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:15:37.345901+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:15:37.345901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:15:47.013 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.021+0000 7fcacbc4a640 10 librados: watch_flush enter 2024-03-27T03:15:47.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.021+0000 7fcacbc4a640 10 librados: watch_flush exit 2024-03-27T03:15:47.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.021+0000 7fcacbc4a640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-27T03:15:47.014 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.021+0000 7fcacbc4a640 10 client.4560.objecter successfully canceled tick 2024-03-27T03:15:47.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.024+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 >> v1:172.21.15.153:6816/2637937781 conn(0x7fcaa806df70 legacy=0x7fcaa8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:47.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.024+0000 7fcacbc4a640 10 monclient: shutdown 2024-03-27T03:15:47.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.024+0000 7fcacbc4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:47.017 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.024+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 >> v1:172.21.15.153:6789/0 conn(0x7fcac40af900 legacy=0x7fcac413c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:47.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.026+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 shutdown_connections 2024-03-27T03:15:47.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.026+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 >> 172.21.15.153:0/401366619 conn(0x7fcac40b35e0 msgr2=0x7fcac413b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:47.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.027+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 shutdown_connections 2024-03-27T03:15:47.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.027+0000 7fcacbc4a640 1 -- 172.21.15.153:0/401366619 wait complete. 2024-03-27T03:15:47.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:47.027+0000 7fcacbc4a640 1 librados: shutdown 2024-03-27T03:15:47.040 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:15:53.846 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:15:54.000 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.007+0000 7fdb445c1640 10 monclient: get_monmap_and_config 2024-03-27T03:15:54.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.007+0000 7fdb445c1640 10 monclient: build_initial_monmap 2024-03-27T03:15:54.001 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.007+0000 7fdb445c1640 10 monclient: monmap: 2024-03-27T03:15:54.001 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:54.001 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:54.001 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:15:54.008524+0000 2024-03-27T03:15:54.001 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:15:54.008524+0000 2024-03-27T03:15:54.001 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:54.001 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:54.001 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:54.001 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:15:54.001 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:15:54.001 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:54.002 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.008+0000 7fdb445c1640 1 Processor -- start 2024-03-27T03:15:54.002 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.008+0000 7fdb445c1640 1 -- start start 2024-03-27T03:15:54.002 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.008+0000 7fdb445c1640 10 monclient: init 2024-03-27T03:15:54.002 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.009+0000 7fdb445c1640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:54.002 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.009+0000 7fdb445c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:15:54.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.013+0000 7fdb445c1640 10 monclient(hunting): picked mon.noname-a con 0x7fdb3c06d9d0 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:54.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.013+0000 7fdb445c1640 10 monclient(hunting): picked mon.noname-c con 0x7fdb3c13e820 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:54.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.013+0000 7fdb445c1640 10 monclient(hunting): picked mon.noname-b con 0x7fdb3c0af1d0 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:54.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.013+0000 7fdb445c1640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0aed10 con 0x7fdb3c06d9d0 2024-03-27T03:15:54.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.013+0000 7fdb445c1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0afd50 con 0x7fdb3c13e820 2024-03-27T03:15:54.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.013+0000 7fdb445c1640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0ab490 con 0x7fdb3c0af1d0 2024-03-27T03:15:54.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.013+0000 7fdb445c1640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:54.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.013+0000 7fdb445c1640 10 monclient(hunting): authenticate will time out at 1914.319345s 2024-03-27T03:15:54.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.013+0000 7fdb4235d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdb3c06d9d0 0x7fdb3c0b05f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:43075/0 (socket says 172.21.15.153:43075) 2024-03-27T03:15:54.007 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.013+0000 7fdb4235d640 1 -- 172.21.15.153:0/3037043606 learned_addr learned my addr 172.21.15.153:0/3037043606 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:54.008 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.014+0000 7fdb4135b640 1 -- 172.21.15.153:0/3037043606 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942490753 0 0) 0x7fdb3c0aed10 con 0x7fdb3c06d9d0 2024-03-27T03:15:54.008 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.014+0000 7fdb4135b640 10 monclient(hunting): _negotiate 2024-03-27T03:15:54.008 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.014+0000 7fdb4135b640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:54.008 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.014+0000 7fdb4135b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:54.008 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.014+0000 7fdb4135b640 10 monclient(hunting): my global_id is 4563 2024-03-27T03:15:54.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.014+0000 7fdb4135b640 1 -- 172.21.15.153:0/3037043606 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb28003520 con 0x7fdb3c06d9d0 2024-03-27T03:15:54.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.015+0000 7fdb4135b640 1 -- 172.21.15.153:0/3037043606 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3759706119 0 0) 0x7fdb28003520 con 0x7fdb3c06d9d0 2024-03-27T03:15:54.009 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.015+0000 7fdb4135b640 1 -- 172.21.15.153:0/3037043606 >> v1:172.21.15.153:6790/0 conn(0x7fdb3c0af1d0 legacy=0x7fdb3c0af640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:15:54.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.020+0000 7fdb4135b640 1 -- 172.21.15.153:0/3037043606 >> v1:172.21.15.196:6789/0 conn(0x7fdb3c13e820 legacy=0x7fdb3c13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:54.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.020+0000 7fdb4135b640 10 monclient: _finish_hunting 0 2024-03-27T03:15:54.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.020+0000 7fdb4135b640 1 monclient: found mon.noname-a 2024-03-27T03:15:54.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.020+0000 7fdb4135b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:54.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.020+0000 7fdb4135b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:15:54.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.020+0000 7fdb4135b640 1 -- 172.21.15.153:0/3037043606 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c0ab770 con 0x7fdb3c06d9d0 2024-03-27T03:15:54.018 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 10 monclient: _finish_auth 0 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 10 monclient: _check_auth_tickets 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 1 -- 172.21.15.153:0/3037043606 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdb38001690 con 0x7fdb3c06d9d0 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 10 monclient: dump: 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:54.019 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb445c1640 5 monclient: authenticate success, global_id 4563 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb445c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 1 -- 172.21.15.153:0/3037043606 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb38001920 con 0x7fdb3c06d9d0 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 1 -- 172.21.15.153:0/3037043606 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdb38001cb0 con 0x7fdb3c06d9d0 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.021+0000 7fdb4135b640 10 monclient: dump: 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:54.020 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:54.021 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:54.021 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:54.021 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:54.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.022+0000 7fdb445c1640 10 monclient: get_monmap_and_config success 2024-03-27T03:15:54.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.022+0000 7fdb445c1640 4 set_mon_vals no callback set 2024-03-27T03:15:54.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.022+0000 7fdb445c1640 10 monclient: shutdown 2024-03-27T03:15:54.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.022+0000 7fdb445c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:54.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.022+0000 7fdb445c1640 1 -- 172.21.15.153:0/3037043606 >> v1:172.21.15.153:6789/0 conn(0x7fdb3c06d9d0 legacy=0x7fdb3c0b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:54.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.023+0000 7fdb445c1640 1 -- 172.21.15.153:0/3037043606 shutdown_connections 2024-03-27T03:15:54.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.023+0000 7fdb445c1640 1 -- 172.21.15.153:0/3037043606 >> 172.21.15.153:0/3037043606 conn(0x7fdb3c06b660 msgr2=0x7fdb3c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:54.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.023+0000 7fdb445c1640 1 -- 172.21.15.153:0/3037043606 shutdown_connections 2024-03-27T03:15:54.021 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.023+0000 7fdb445c1640 1 -- 172.21.15.153:0/3037043606 wait complete. 2024-03-27T03:15:54.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.027+0000 7fdb445c1640 10 monclient: build_initial_monmap 2024-03-27T03:15:54.022 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.027+0000 7fdb445c1640 10 monclient: monmap: 2024-03-27T03:15:54.022 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:15:54.022 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:15:54.022 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.028+0000 7fdb445c1640 1 librados: starting msgr at 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.028+0000 7fdb445c1640 1 librados: starting objecter 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.028+0000 7fdb445c1640 1 Processor -- start 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.028+0000 7fdb445c1640 1 -- start start 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.028+0000 7fdb445c1640 1 librados: setting wanted keys 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.028+0000 7fdb445c1640 1 librados: calling monclient init 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.028+0000 7fdb445c1640 10 monclient: init 2024-03-27T03:15:54.023 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.029+0000 7fdb445c1640 10 monclient: _reopen_session rank -1 2024-03-27T03:15:54.024 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.029+0000 7fdb445c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:15:54.027 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.033+0000 7fdb445c1640 10 monclient(hunting): picked mon.noname-b con 0x7fdb3c06d9d0 addr v1:172.21.15.196:6789/0 2024-03-27T03:15:54.027 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.033+0000 7fdb445c1640 10 monclient(hunting): picked mon.noname-c con 0x7fdb3c0af1d0 addr v1:172.21.15.153:6790/0 2024-03-27T03:15:54.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.033+0000 7fdb445c1640 10 monclient(hunting): picked mon.noname-a con 0x7fdb3c13e820 addr v1:172.21.15.153:6789/0 2024-03-27T03:15:54.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.033+0000 7fdb445c1640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c06f960 con 0x7fdb3c13e820 2024-03-27T03:15:54.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.033+0000 7fdb445c1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c071ba0 con 0x7fdb3c06d9d0 2024-03-27T03:15:54.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.033+0000 7fdb445c1640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c06fbb0 con 0x7fdb3c0af1d0 2024-03-27T03:15:54.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.033+0000 7fdb445c1640 10 monclient(hunting): _renew_subs 2024-03-27T03:15:54.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.033+0000 7fdb445c1640 10 monclient(hunting): authenticate will time out at 1914.338821s 2024-03-27T03:15:54.028 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.033+0000 7fdb4235d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdb3c06d9d0 0x7fdb3c06e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:33179/0 (socket says 172.21.15.153:33179) 2024-03-27T03:15:54.029 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.033+0000 7fdb4235d640 1 -- 172.21.15.153:0/258421753 learned_addr learned my addr 172.21.15.153:0/258421753 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:15:54.036 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.037+0000 7fdb32ffd640 10 client.?.objecter ms_handle_connect 0x7fdb3c06d9d0 2024-03-27T03:15:54.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.037+0000 7fdb32ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:54.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.037+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416965335 0 0) 0x7fdb3c071ba0 con 0x7fdb3c06d9d0 2024-03-27T03:15:54.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.037+0000 7fdb32ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:15:54.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.037+0000 7fdb32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:54.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.037+0000 7fdb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:54.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 10 monclient(hunting): my global_id is 4597 2024-03-27T03:15:54.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb20004eb0 con 0x7fdb3c06d9d0 2024-03-27T03:15:54.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 10 client.?.objecter ms_handle_connect 0x7fdb3c13e820 2024-03-27T03:15:54.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:54.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 10 client.?.objecter ms_handle_connect 0x7fdb3c0af1d0 2024-03-27T03:15:54.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:15:54.037 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287113593 0 0) 0x7fdb3c06f960 con 0x7fdb3c13e820 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 10 monclient(hunting): my global_id is 4566 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb3c071ba0 con 0x7fdb3c13e820 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704796736 0 0) 0x7fdb3c06fbb0 con 0x7fdb3c0af1d0 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.038+0000 7fdb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 10 monclient(hunting): my global_id is 4658 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb3c06f960 con 0x7fdb3c0af1d0 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2145313607 0 0) 0x7fdb3c071ba0 con 0x7fdb3c13e820 2024-03-27T03:15:54.038 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb3c06fbb0 con 0x7fdb3c13e820 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdb34001a70 con 0x7fdb3c13e820 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 10 monclient(hunting): dump: 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:54.039 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3294408937 0 0) 0x7fdb3c06f960 con 0x7fdb3c0af1d0 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb3c071ba0 con 0x7fdb3c0af1d0 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4016236684 0 0) 0x7fdb3c06fbb0 con 0x7fdb3c13e820 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 >> v1:172.21.15.153:6790/0 conn(0x7fdb3c0af1d0 legacy=0x7fdb3c134730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.039+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 >> v1:172.21.15.196:6789/0 conn(0x7fdb3c06d9d0 legacy=0x7fdb3c06e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb32ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb32ffd640 1 monclient: found mon.a 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb32ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c06fe90 con 0x7fdb3c13e820 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb32ffd640 10 monclient: _finish_auth 0 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb32ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:15:54.040 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb34001f60 con 0x7fdb3c13e820 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb445c1640 5 monclient: authenticate success, global_id 4566 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb445c1640 10 monclient: _renew_subs 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb445c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb445c1640 1 -- 172.21.15.153:0/258421753 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c06efc0 con 0x7fdb3c13e820 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb445c1640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb445c1640 10 monclient: _renew_subs 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.040+0000 7fdb445c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.041+0000 7fdb445c1640 1 -- 172.21.15.153:0/258421753 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c135020 con 0x7fdb3c13e820 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.041+0000 7fdb445c1640 1 librados: init done 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.041+0000 7fdb32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.041+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fdb34003b10 con 0x7fdb3c13e820 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.041+0000 7fdb32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.041+0000 7fdb32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.041+0000 7fdb32ffd640 10 monclient: dump: 2024-03-27T03:15:54.041 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:15:54.042 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:15:54.042 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:54.042 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:15:54.042 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:15:54.042 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:15:54.042 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:15:54.042 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:15:54.042 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:15:54.042 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:15:54.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.043+0000 7fdb33fff640 4 set_mon_vals no callback set 2024-03-27T03:15:54.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.043+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fdb34019ae0 con 0x7fdb3c13e820 2024-03-27T03:15:54.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.043+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2380031179 0 0) 0x7fdb34083980 con 0x7fdb3c13e820 2024-03-27T03:15:54.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.043+0000 7fdb32ffd640 10 client.4566.objecter ms_dispatch 0x7fdb3c000b90 osd_map(11..11 src has 1..11) v4 2024-03-27T03:15:54.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.043+0000 7fdb32ffd640 3 client.4566.objecter handle_osd_map got epochs [11,11] > 0 2024-03-27T03:15:54.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.043+0000 7fdb32ffd640 3 client.4566.objecter handle_osd_map decoding full epoch 11 2024-03-27T03:15:54.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.043+0000 7fdb4235d640 10 monclient: get_auth_request con 0x7fdb2006e010 auth_method 0 2024-03-27T03:15:54.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.044+0000 7fdb32ffd640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-27T03:15:54.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.044+0000 7fdb32ffd640 10 client.4566.objecter ms_handle_connect 0x7fdb2006e010 2024-03-27T03:15:54.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.044+0000 7fdb445c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:54.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.044+0000 7fdb445c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:54.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.044+0000 7fdb445c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:54.044 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.044+0000 7fdb445c1640 1 -- 172.21.15.153:0/258421753 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb24005080 con 0x7fdb3c13e820 2024-03-27T03:15:54.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.052+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb3404f260 con 0x7fdb3c13e820 2024-03-27T03:15:54.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.052+0000 7fdb32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:15:54.046 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.052+0000 7fdb32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:15:54.365 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.370+0000 7fdb445c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:54.365 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.370+0000 7fdb445c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:54.365 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.370+0000 7fdb445c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:15:54.365 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.370+0000 7fdb445c1640 1 -- 172.21.15.153:0/258421753 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb24005330 con 0x7fdb3c13e820 2024-03-27T03:15:54.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.376+0000 7fdb32ffd640 1 -- 172.21.15.153:0/258421753 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 992601923) 0x7fdb34051190 con 0x7fdb3c13e820 2024-03-27T03:15:54.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.376+0000 7fdb32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:15:54.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.376+0000 7fdb32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:15:54.369 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:15:54.370 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":11,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:15:37.345901+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:15:37.345901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:15:54.373 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.380+0000 7fdb445c1640 10 librados: watch_flush enter 2024-03-27T03:15:54.373 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.380+0000 7fdb445c1640 10 librados: watch_flush exit 2024-03-27T03:15:54.373 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.380+0000 7fdb445c1640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-27T03:15:54.373 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.380+0000 7fdb445c1640 10 client.4566.objecter successfully canceled tick 2024-03-27T03:15:54.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.381+0000 7fdb445c1640 1 -- 172.21.15.153:0/258421753 >> v1:172.21.15.153:6816/2637937781 conn(0x7fdb2006e010 legacy=0x7fdb200704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:54.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.381+0000 7fdb445c1640 10 monclient: shutdown 2024-03-27T03:15:54.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.381+0000 7fdb445c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:15:54.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.381+0000 7fdb445c1640 1 -- 172.21.15.153:0/258421753 >> v1:172.21.15.153:6789/0 conn(0x7fdb3c13e820 legacy=0x7fdb3c070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:15:54.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.381+0000 7fdb445c1640 1 -- 172.21.15.153:0/258421753 shutdown_connections 2024-03-27T03:15:54.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.381+0000 7fdb445c1640 1 -- 172.21.15.153:0/258421753 >> 172.21.15.153:0/258421753 conn(0x7fdb3c06b660 msgr2=0x7fdb3c13ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:15:54.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.382+0000 7fdb445c1640 1 -- 172.21.15.153:0/258421753 shutdown_connections 2024-03-27T03:15:54.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.382+0000 7fdb445c1640 1 -- 172.21.15.153:0/258421753 wait complete. 2024-03-27T03:15:54.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:15:54.382+0000 7fdb445c1640 1 librados: shutdown 2024-03-27T03:15:54.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:16:01.202 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:16:01.364 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.370+0000 7f18e86a8640 10 monclient: get_monmap_and_config 2024-03-27T03:16:01.365 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.370+0000 7f18e86a8640 10 monclient: build_initial_monmap 2024-03-27T03:16:01.365 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.370+0000 7f18e86a8640 10 monclient: monmap: 2024-03-27T03:16:01.365 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:16:01.365 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:01.365 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:16:01.371789+0000 2024-03-27T03:16:01.365 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:16:01.371789+0000 2024-03-27T03:16:01.365 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:16:01.365 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:01.366 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:16:01.366 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:16:01.366 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:16:01.366 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:01.367 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.372+0000 7f18e86a8640 1 Processor -- start 2024-03-27T03:16:01.367 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.372+0000 7f18e86a8640 1 -- start start 2024-03-27T03:16:01.367 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.372+0000 7f18e86a8640 10 monclient: init 2024-03-27T03:16:01.367 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.373+0000 7f18e86a8640 10 monclient: _reopen_session rank -1 2024-03-27T03:16:01.367 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.373+0000 7f18e86a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:16:01.367 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.373+0000 7f18e86a8640 10 monclient(hunting): picked mon.noname-a con 0x7f18e00a9930 addr v1:172.21.15.153:6789/0 2024-03-27T03:16:01.367 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.373+0000 7f18e86a8640 10 monclient(hunting): picked mon.noname-b con 0x7f18e0149dc0 addr v1:172.21.15.153:6790/0 2024-03-27T03:16:01.369 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.373+0000 7f18e86a8640 10 monclient(hunting): picked mon.noname-c con 0x7f18e014ff40 addr v1:172.21.15.196:6789/0 2024-03-27T03:16:01.370 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.376+0000 7f18e86a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00aae60 con 0x7f18e00a9930 2024-03-27T03:16:01.370 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.376+0000 7f18e86a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00b4310 con 0x7f18e014ff40 2024-03-27T03:16:01.370 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.376+0000 7f18e86a8640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00afdd0 con 0x7f18e0149dc0 2024-03-27T03:16:01.370 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.376+0000 7f18e86a8640 10 monclient(hunting): _renew_subs 2024-03-27T03:16:01.370 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.376+0000 7f18e86a8640 10 monclient(hunting): authenticate will time out at 1921.682364s 2024-03-27T03:16:01.374 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.378+0000 7f18e6444640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f18e00a9930 0x7f18e0143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:49807/0 (socket says 172.21.15.153:49807) 2024-03-27T03:16:01.374 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.378+0000 7f18e6444640 1 -- 172.21.15.153:0/702197711 learned_addr learned my addr 172.21.15.153:0/702197711 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:16:01.374 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.380+0000 7f18e5442640 1 -- 172.21.15.153:0/702197711 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779972796 0 0) 0x7f18e00aae60 con 0x7f18e00a9930 2024-03-27T03:16:01.374 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.380+0000 7f18e5442640 10 monclient(hunting): _negotiate 2024-03-27T03:16:01.374 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.380+0000 7f18e5442640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:01.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.380+0000 7f18e5442640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:01.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.380+0000 7f18e5442640 10 monclient(hunting): my global_id is 4569 2024-03-27T03:16:01.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.380+0000 7f18e5442640 1 -- 172.21.15.153:0/702197711 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18cc003520 con 0x7f18e00a9930 2024-03-27T03:16:01.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.381+0000 7f18e5442640 1 -- 172.21.15.153:0/702197711 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3307460156 0 0) 0x7f18cc003520 con 0x7f18e00a9930 2024-03-27T03:16:01.375 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.381+0000 7f18e5442640 1 -- 172.21.15.153:0/702197711 >> v1:172.21.15.153:6790/0 conn(0x7f18e0149dc0 legacy=0x7f18e014a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T03:16:01.376 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 1 -- 172.21.15.153:0/702197711 >> v1:172.21.15.196:6789/0 conn(0x7f18e014ff40 legacy=0x7f18e0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 10 monclient: _finish_hunting 0 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 1 monclient: found mon.noname-a 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 1 -- 172.21.15.153:0/702197711 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e014a950 con 0x7f18e00a9930 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 10 monclient: _finish_auth 0 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 10 monclient: _check_auth_tickets 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 1 -- 172.21.15.153:0/702197711 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f18d8001a10 con 0x7f18e00a9930 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e5442640 10 monclient: dump: 2024-03-27T03:16:01.377 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e86a8640 5 monclient: authenticate success, global_id 4569 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.382+0000 7f18e86a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.383+0000 7f18e5442640 1 -- 172.21.15.153:0/702197711 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d8001be0 con 0x7f18e00a9930 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.383+0000 7f18e5442640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.383+0000 7f18e5442640 1 -- 172.21.15.153:0/702197711 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f18d8002030 con 0x7f18e00a9930 2024-03-27T03:16:01.378 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.383+0000 7f18e5442640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.383+0000 7f18e5442640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.383+0000 7f18e5442640 10 monclient: dump: 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.383+0000 7f18e86a8640 10 monclient: get_monmap_and_config success 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.383+0000 7f18e86a8640 4 set_mon_vals no callback set 2024-03-27T03:16:01.379 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.384+0000 7f18e86a8640 10 monclient: shutdown 2024-03-27T03:16:01.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.384+0000 7f18e86a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:16:01.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.384+0000 7f18e86a8640 1 -- 172.21.15.153:0/702197711 >> v1:172.21.15.153:6789/0 conn(0x7f18e00a9930 legacy=0x7f18e0143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:01.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.384+0000 7f18e86a8640 1 -- 172.21.15.153:0/702197711 shutdown_connections 2024-03-27T03:16:01.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.384+0000 7f18e86a8640 1 -- 172.21.15.153:0/702197711 >> 172.21.15.153:0/702197711 conn(0x7f18e00b35e0 msgr2=0x7f18e00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:16:01.380 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.386+0000 7f18e86a8640 1 -- 172.21.15.153:0/702197711 shutdown_connections 2024-03-27T03:16:01.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.388+0000 7f18e86a8640 1 -- 172.21.15.153:0/702197711 wait complete. 2024-03-27T03:16:01.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.389+0000 7f18e86a8640 10 monclient: build_initial_monmap 2024-03-27T03:16:01.382 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.389+0000 7f18e86a8640 10 monclient: monmap: 2024-03-27T03:16:01.383 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:16:01.383 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:16:01.383 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:16:01.383 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:16:01.383 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:16:01.383 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:01.383 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:16:01.383 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:16:01.383 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:16:01.383 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:01.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.390+0000 7f18e86a8640 1 librados: starting msgr at 2024-03-27T03:16:01.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.390+0000 7f18e86a8640 1 librados: starting objecter 2024-03-27T03:16:01.384 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.390+0000 7f18e86a8640 1 Processor -- start 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.391+0000 7f18e86a8640 1 -- start start 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.391+0000 7f18e86a8640 1 librados: setting wanted keys 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.391+0000 7f18e86a8640 1 librados: calling monclient init 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.391+0000 7f18e86a8640 10 monclient: init 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.392+0000 7f18e86a8640 10 monclient: _reopen_session rank -1 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.392+0000 7f18e86a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.392+0000 7f18e86a8640 10 monclient(hunting): picked mon.noname-c con 0x7f18e00a9930 addr v1:172.21.15.153:6790/0 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.392+0000 7f18e86a8640 10 monclient(hunting): picked mon.noname-b con 0x7f18e0149dc0 addr v1:172.21.15.196:6789/0 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.392+0000 7f18e86a8640 10 monclient(hunting): picked mon.noname-a con 0x7f18e014ff40 addr v1:172.21.15.153:6789/0 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.392+0000 7f18e86a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e006ee70 con 0x7f18e014ff40 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.392+0000 7f18e86a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e01db380 con 0x7f18e0149dc0 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.392+0000 7f18e86a8640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e01dc250 con 0x7f18e00a9930 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.392+0000 7f18e86a8640 10 monclient(hunting): _renew_subs 2024-03-27T03:16:01.389 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.392+0000 7f18e86a8640 10 monclient(hunting): authenticate will time out at 1921.697965s 2024-03-27T03:16:01.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.396+0000 7f18e6c45640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f18e014ff40 0x7f18e01e5dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:49907/0 (socket says 172.21.15.153:49907) 2024-03-27T03:16:01.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.396+0000 7f18e6444640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f18e00a9930 0x7f18e013f220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:34297/0 (socket says 172.21.15.153:34297) 2024-03-27T03:16:01.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.396+0000 7f18e6444640 1 -- 172.21.15.153:0/2429100143 learned_addr learned my addr 172.21.15.153:0/2429100143 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:16:01.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 10 client.?.objecter ms_handle_connect 0x7f18e014ff40 2024-03-27T03:16:01.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:16:01.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 10 client.?.objecter ms_handle_connect 0x7f18e00a9930 2024-03-27T03:16:01.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:16:01.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370466413 0 0) 0x7f18e006ee70 con 0x7f18e014ff40 2024-03-27T03:16:01.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 10 monclient(hunting): _negotiate 2024-03-27T03:16:01.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 10 monclient(hunting): my global_id is 4572 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c4004eb0 con 0x7f18e014ff40 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4199438568 0 0) 0x7f18c4004eb0 con 0x7f18e014ff40 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18e006ee70 con 0x7f18e014ff40 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f18d0003ea0 con 0x7f18e014ff40 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 10 monclient(hunting): dump: 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:01.396 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.397+0000 7f18d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 10 client.?.objecter ms_handle_connect 0x7f18e0149dc0 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2671617938 0 0) 0x7f18e006ee70 con 0x7f18e014ff40 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 >> v1:172.21.15.153:6790/0 conn(0x7f18e00a9930 legacy=0x7f18e013f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 >> v1:172.21.15.196:6789/0 conn(0x7f18e0149dc0 legacy=0x7f18e01dbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 10 monclient: _finish_hunting 0 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 1 monclient: found mon.a 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:01.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e01db620 con 0x7f18e014ff40 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 10 monclient: _finish_auth 0 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 10 monclient: _check_auth_tickets 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d0004430 con 0x7f18e014ff40 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f18d0001070 con 0x7f18e014ff40 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d6ffd640 10 monclient: dump: 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:01.398 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:01.399 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:01.399 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:01.399 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:01.399 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:01.399 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:01.399 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:01.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.398+0000 7f18d7fff640 4 set_mon_vals no callback set 2024-03-27T03:16:01.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.399+0000 7f18e86a8640 5 monclient: authenticate success, global_id 4572 2024-03-27T03:16:01.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.399+0000 7f18e86a8640 10 monclient: _renew_subs 2024-03-27T03:16:01.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.399+0000 7f18e86a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:01.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.399+0000 7f18e86a8640 1 -- 172.21.15.153:0/2429100143 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e013e4a0 con 0x7f18e014ff40 2024-03-27T03:16:01.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.399+0000 7f18e86a8640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:16:01.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.399+0000 7f18e86a8640 10 monclient: _renew_subs 2024-03-27T03:16:01.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.399+0000 7f18e86a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:01.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.399+0000 7f18e86a8640 1 -- 172.21.15.153:0/2429100143 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e00ab560 con 0x7f18e014ff40 2024-03-27T03:16:01.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.399+0000 7f18e86a8640 1 librados: init done 2024-03-27T03:16:01.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.401+0000 7f18e86a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:01.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.401+0000 7f18e86a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:01.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.401+0000 7f18e86a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:01.400 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.401+0000 7f18e86a8640 1 -- 172.21.15.153:0/2429100143 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e013e7a0 con 0x7f18e014ff40 2024-03-27T03:16:01.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.405+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f18d00011d0 con 0x7f18e014ff40 2024-03-27T03:16:01.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.406+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2380031179 0 0) 0x7f18d00846b0 con 0x7f18e014ff40 2024-03-27T03:16:01.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.406+0000 7f18d6ffd640 10 client.4572.objecter ms_dispatch 0x7f18e0000b90 osd_map(11..11 src has 1..11) v4 2024-03-27T03:16:01.401 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.406+0000 7f18d6ffd640 3 client.4572.objecter handle_osd_map got epochs [11,11] > 0 2024-03-27T03:16:01.402 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.406+0000 7f18d6ffd640 3 client.4572.objecter handle_osd_map decoding full epoch 11 2024-03-27T03:16:01.402 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.406+0000 7f18d6ffd640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-27T03:16:01.402 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.406+0000 7f18e6444640 10 monclient: get_auth_request con 0x7f18c406dcb0 auth_method 0 2024-03-27T03:16:01.402 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.406+0000 7f18d6ffd640 10 client.4572.objecter ms_handle_connect 0x7f18c406dcb0 2024-03-27T03:16:01.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.413+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18d004ff90 con 0x7f18e014ff40 2024-03-27T03:16:01.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.413+0000 7f18d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:01.407 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.413+0000 7f18d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:16:01.730 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.735+0000 7f18e86a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:01.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.736+0000 7f18e86a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:01.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.736+0000 7f18e86a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:01.731 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.736+0000 7f18e86a8640 1 -- 172.21.15.153:0/2429100143 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18e013e7a0 con 0x7f18e014ff40 2024-03-27T03:16:01.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.739+0000 7f18d6ffd640 1 -- 172.21.15.153:0/2429100143 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 992601923) 0x7f18e013e7a0 con 0x7f18e014ff40 2024-03-27T03:16:01.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.739+0000 7f18d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:01.732 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.739+0000 7f18d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:16:01.732 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:16:01.732 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":11,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:15:37.345901+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:15:37.345901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:16:01.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.742+0000 7f18e86a8640 10 librados: watch_flush enter 2024-03-27T03:16:01.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.745+0000 7f18e86a8640 10 librados: watch_flush exit 2024-03-27T03:16:01.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.745+0000 7f18e86a8640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-27T03:16:01.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.745+0000 7f18e86a8640 10 client.4572.objecter successfully canceled tick 2024-03-27T03:16:01.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.745+0000 7f18e86a8640 1 -- 172.21.15.153:0/2429100143 >> v1:172.21.15.153:6816/2637937781 conn(0x7f18c406dcb0 legacy=0x7f18c4070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:01.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.745+0000 7f18e86a8640 10 monclient: shutdown 2024-03-27T03:16:01.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.745+0000 7f18e86a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:16:01.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.745+0000 7f18e86a8640 1 -- 172.21.15.153:0/2429100143 >> v1:172.21.15.153:6789/0 conn(0x7f18e014ff40 legacy=0x7f18e01e5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:01.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.746+0000 7f18e86a8640 1 -- 172.21.15.153:0/2429100143 shutdown_connections 2024-03-27T03:16:01.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.746+0000 7f18e86a8640 1 -- 172.21.15.153:0/2429100143 >> 172.21.15.153:0/2429100143 conn(0x7f18e00b35e0 msgr2=0x7f18e00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:16:01.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.746+0000 7f18e86a8640 1 -- 172.21.15.153:0/2429100143 shutdown_connections 2024-03-27T03:16:01.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.746+0000 7f18e86a8640 1 -- 172.21.15.153:0/2429100143 wait complete. 2024-03-27T03:16:01.740 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:01.747+0000 7f18e86a8640 1 librados: shutdown 2024-03-27T03:16:01.758 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:16:08.566 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:16:08.712 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.718+0000 7fde2ea74640 10 monclient: get_monmap_and_config 2024-03-27T03:16:08.712 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.718+0000 7fde2ea74640 10 monclient: build_initial_monmap 2024-03-27T03:16:08.712 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.718+0000 7fde2ea74640 10 monclient: monmap: 2024-03-27T03:16:08.712 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:16:08.712 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:08.712 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:16:08.719817+0000 2024-03-27T03:16:08.712 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:16:08.719817+0000 2024-03-27T03:16:08.713 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:16:08.713 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:08.713 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:16:08.713 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:16:08.713 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:16:08.713 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:08.714 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.719+0000 7fde2ea74640 1 Processor -- start 2024-03-27T03:16:08.714 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.719+0000 7fde2ea74640 1 -- start start 2024-03-27T03:16:08.714 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.719+0000 7fde2ea74640 10 monclient: init 2024-03-27T03:16:08.714 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.720+0000 7fde2ea74640 10 monclient: _reopen_session rank -1 2024-03-27T03:16:08.714 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.720+0000 7fde2ea74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:16:08.714 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.720+0000 7fde2ea74640 10 monclient(hunting): picked mon.noname-a con 0x7fde281457f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:16:08.714 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.720+0000 7fde2ea74640 10 monclient(hunting): picked mon.noname-c con 0x7fde2813f650 addr v1:172.21.15.196:6789/0 2024-03-27T03:16:08.714 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.720+0000 7fde2ea74640 10 monclient(hunting): picked mon.noname-b con 0x7fde280a9d70 addr v1:172.21.15.153:6790/0 2024-03-27T03:16:08.714 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.720+0000 7fde2ea74640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde280aaef0 con 0x7fde281457f0 2024-03-27T03:16:08.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.720+0000 7fde2ea74640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde28145bd0 con 0x7fde2813f650 2024-03-27T03:16:08.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.720+0000 7fde2ea74640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde28145eb0 con 0x7fde280a9d70 2024-03-27T03:16:08.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.720+0000 7fde2ea74640 10 monclient(hunting): _renew_subs 2024-03-27T03:16:08.715 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.720+0000 7fde2ea74640 10 monclient(hunting): authenticate will time out at 1929.026482s 2024-03-27T03:16:08.718 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde27fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fde2813f650 0x7fde280b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:49504/0 (socket says 172.21.15.153:49504) 2024-03-27T03:16:08.718 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde27fff640 1 -- 172.21.15.153:0/3082181002 learned_addr learned my addr 172.21.15.153:0/3082181002 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde277fe640 1 -- 172.21.15.153:0/3082181002 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075120135 0 0) 0x7fde280aaef0 con 0x7fde281457f0 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde277fe640 10 monclient(hunting): _negotiate 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde277fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde277fe640 10 monclient(hunting): my global_id is 4575 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde277fe640 1 -- 172.21.15.153:0/3082181002 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde18003520 con 0x7fde281457f0 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde277fe640 1 -- 172.21.15.153:0/3082181002 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854563203 0 0) 0x7fde28145bd0 con 0x7fde2813f650 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde277fe640 10 monclient(hunting): _negotiate 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde277fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde277fe640 10 monclient(hunting): my global_id is 4603 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.723+0000 7fde277fe640 1 -- 172.21.15.153:0/3082181002 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde280aaef0 con 0x7fde2813f650 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 1 -- 172.21.15.153:0/3082181002 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 226370531 0 0) 0x7fde18003520 con 0x7fde281457f0 2024-03-27T03:16:08.719 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 1 -- 172.21.15.153:0/3082181002 >> v1:172.21.15.153:6790/0 conn(0x7fde280a9d70 legacy=0x7fde2814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 1 -- 172.21.15.153:0/3082181002 >> v1:172.21.15.196:6789/0 conn(0x7fde2813f650 legacy=0x7fde280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 10 monclient: _finish_hunting 0 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 1 monclient: found mon.noname-a 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 1 -- 172.21.15.153:0/3082181002 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde28146340 con 0x7fde281457f0 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 10 monclient: _finish_auth 0 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 10 monclient: _check_auth_tickets 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde2ea74640 5 monclient: authenticate success, global_id 4575 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde2ea74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 1 -- 172.21.15.153:0/3082181002 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fde1c001690 con 0x7fde281457f0 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 10 monclient: dump: 2024-03-27T03:16:08.720 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde2ea74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 1 -- 172.21.15.153:0/3082181002 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde1c0018b0 con 0x7fde281457f0 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 1 -- 172.21.15.153:0/3082181002 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fde1c001c40 con 0x7fde281457f0 2024-03-27T03:16:08.721 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde277fe640 10 monclient: dump: 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde2ea74640 10 monclient: get_monmap_and_config success 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.724+0000 7fde2ea74640 4 set_mon_vals no callback set 2024-03-27T03:16:08.722 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.725+0000 7fde2ea74640 10 monclient: shutdown 2024-03-27T03:16:08.723 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.725+0000 7fde2ea74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:16:08.723 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.725+0000 7fde2ea74640 1 -- 172.21.15.153:0/3082181002 >> v1:172.21.15.153:6789/0 conn(0x7fde281457f0 legacy=0x7fde2813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:08.723 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.725+0000 7fde2ea74640 1 -- 172.21.15.153:0/3082181002 shutdown_connections 2024-03-27T03:16:08.723 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.725+0000 7fde2ea74640 1 -- 172.21.15.153:0/3082181002 >> 172.21.15.153:0/3082181002 conn(0x7fde28066a00 msgr2=0x7fde28066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:16:08.723 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.725+0000 7fde2ea74640 1 -- 172.21.15.153:0/3082181002 shutdown_connections 2024-03-27T03:16:08.723 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.725+0000 7fde2ea74640 1 -- 172.21.15.153:0/3082181002 wait complete. 2024-03-27T03:16:08.725 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.726+0000 7fde2ea74640 10 monclient: build_initial_monmap 2024-03-27T03:16:08.725 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.726+0000 7fde2ea74640 10 monclient: monmap: 2024-03-27T03:16:08.725 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:16:08.725 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:16:08.725 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:16:08.725 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.727+0000 7fde2ea74640 1 librados: starting msgr at 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.727+0000 7fde2ea74640 1 librados: starting objecter 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.727+0000 7fde2ea74640 1 Processor -- start 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.727+0000 7fde2ea74640 1 -- start start 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.727+0000 7fde2ea74640 1 librados: setting wanted keys 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.727+0000 7fde2ea74640 1 librados: calling monclient init 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.727+0000 7fde2ea74640 10 monclient: init 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.728+0000 7fde2ea74640 10 monclient: _reopen_session rank -1 2024-03-27T03:16:08.726 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.728+0000 7fde2ea74640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.728+0000 7fde2ea74640 10 monclient(hunting): picked mon.noname-c con 0x7fde280a9d70 addr v1:172.21.15.153:6790/0 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.728+0000 7fde2ea74640 10 monclient(hunting): picked mon.noname-b con 0x7fde2813f650 addr v1:172.21.15.196:6789/0 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.729+0000 7fde2ea74640 10 monclient(hunting): picked mon.noname-a con 0x7fde281457f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.729+0000 7fde2ea74640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde280aca60 con 0x7fde281457f0 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.729+0000 7fde2ea74640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde280acd40 con 0x7fde2813f650 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.729+0000 7fde2ea74640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2813fe40 con 0x7fde280a9d70 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.729+0000 7fde2ea74640 10 monclient(hunting): _renew_subs 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.729+0000 7fde2ea74640 10 monclient(hunting): authenticate will time out at 1929.034966s 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.730+0000 7fde27fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fde2813f650 0x7fde28140a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:49630/0 (socket says 172.21.15.153:49630) 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.730+0000 7fde27fff640 1 -- 172.21.15.153:0/3223686157 learned_addr learned my addr 172.21.15.153:0/3223686157 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.731+0000 7fde257fa640 10 client.?.objecter ms_handle_connect 0x7fde2813f650 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.731+0000 7fde257fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.732+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052806737 0 0) 0x7fde280acd40 con 0x7fde2813f650 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.732+0000 7fde257fa640 10 monclient(hunting): _negotiate 2024-03-27T03:16:08.727 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.732+0000 7fde257fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:08.728 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.732+0000 7fde257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:08.728 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.732+0000 7fde257fa640 10 monclient(hunting): my global_id is 4606 2024-03-27T03:16:08.728 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.732+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde0c004eb0 con 0x7fde2813f650 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 10 client.?.objecter ms_handle_connect 0x7fde280a9d70 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 10 client.?.objecter resend_mon_ops 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3907589063 0 0) 0x7fde0c004eb0 con 0x7fde2813f650 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde280acd40 con 0x7fde2813f650 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046876224 0 0) 0x7fde2813fe40 con 0x7fde280a9d70 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 10 monclient(hunting): _negotiate 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 10 monclient(hunting): my global_id is 4664 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde0c004eb0 con 0x7fde280a9d70 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fde140019f0 con 0x7fde2813f650 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:08.733 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.733+0000 7fde257fa640 10 monclient(hunting): dump: 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4042481029 0 0) 0x7fde0c004eb0 con 0x7fde280a9d70 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde2813fe40 con 0x7fde280a9d70 2024-03-27T03:16:08.734 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fde1c001bc0 con 0x7fde280a9d70 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 10 monclient(hunting): dump: 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.2 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1725748712 0 0) 0x7fde2813fe40 con 0x7fde280a9d70 2024-03-27T03:16:08.735 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 >> v1:172.21.15.196:6789/0 conn(0x7fde2813f650 legacy=0x7fde28140a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 >> v1:172.21.15.153:6789/0 conn(0x7fde281457f0 legacy=0x7fde281e1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 10 monclient: _finish_hunting 0 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 1 monclient: found mon.c 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde28137dd0 con 0x7fde280a9d70 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 10 monclient: _finish_auth 0 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 10 monclient: _check_auth_tickets 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde2ea74640 5 monclient: authenticate success, global_id 4664 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde2ea74640 10 monclient: _renew_subs 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde2ea74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde2ea74640 1 -- 172.21.15.153:0/3223686157 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde28142c40 con 0x7fde280a9d70 2024-03-27T03:16:08.736 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.2 v1:172.21.15.153:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde1c000fe0 con 0x7fde280a9d70 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde2ea74640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.734+0000 7fde26ffd640 4 set_mon_vals no callback set 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.735+0000 7fde2ea74640 10 monclient: _renew_subs 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.735+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.2 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7fde1c003a40 con 0x7fde280a9d70 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.735+0000 7fde2ea74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.735+0000 7fde2ea74640 1 -- 172.21.15.153:0/3223686157 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde280aba50 con 0x7fde280a9d70 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.735+0000 7fde2ea74640 1 librados: init done 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.735+0000 7fde257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.735+0000 7fde257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.735+0000 7fde257fa640 10 monclient: dump: 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:08.737 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.736+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.2 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7fde1c019a10 con 0x7fde280a9d70 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.736+0000 7fde2ea74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.737+0000 7fde2ea74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.737+0000 7fde2ea74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.737+0000 7fde2ea74640 1 -- 172.21.15.153:0/3223686157 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde28137fc0 con 0x7fde280a9d70 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.737+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.2 v1:172.21.15.153:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2380031179 0 0) 0x7fde1c084850 con 0x7fde280a9d70 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.737+0000 7fde257fa640 10 client.4664.objecter ms_dispatch 0x7fde28000b90 osd_map(11..11 src has 1..11) v4 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.737+0000 7fde257fa640 3 client.4664.objecter handle_osd_map got epochs [11,11] > 0 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.737+0000 7fde257fa640 3 client.4664.objecter handle_osd_map decoding full epoch 11 2024-03-27T03:16:08.738 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.737+0000 7fde257fa640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-27T03:16:08.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.738+0000 7fde27fff640 10 monclient: get_auth_request con 0x7fde0c06de50 auth_method 0 2024-03-27T03:16:08.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.738+0000 7fde257fa640 10 client.4664.objecter ms_handle_connect 0x7fde0c06de50 2024-03-27T03:16:08.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.746+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.2 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde1c04f100 con 0x7fde280a9d70 2024-03-27T03:16:08.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.746+0000 7fde257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:08.739 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:08.746+0000 7fde257fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:16:09.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.070+0000 7fde2ea74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:09.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.070+0000 7fde2ea74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:09.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.070+0000 7fde2ea74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.153:6790/0 2024-03-27T03:16:09.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.070+0000 7fde2ea74640 1 -- 172.21.15.153:0/3223686157 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde28075f10 con 0x7fde280a9d70 2024-03-27T03:16:09.065 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.072+0000 7fde257fa640 1 -- 172.21.15.153:0/3223686157 <== mon.2 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 992601923) 0x7fde1c051030 con 0x7fde280a9d70 2024-03-27T03:16:09.065 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.072+0000 7fde257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:09.065 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.072+0000 7fde257fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:16:09.065 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:16:09.065 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":11,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:15:37.345901+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:15:37.345901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:16:09.069 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.075+0000 7fde2ea74640 10 librados: watch_flush enter 2024-03-27T03:16:09.069 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.075+0000 7fde2ea74640 10 librados: watch_flush exit 2024-03-27T03:16:09.069 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.075+0000 7fde2ea74640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-27T03:16:09.069 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.075+0000 7fde2ea74640 10 client.4664.objecter successfully canceled tick 2024-03-27T03:16:09.069 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.076+0000 7fde2ea74640 1 -- 172.21.15.153:0/3223686157 >> v1:172.21.15.153:6816/2637937781 conn(0x7fde0c06de50 legacy=0x7fde0c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:09.069 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.076+0000 7fde2ea74640 10 monclient: shutdown 2024-03-27T03:16:09.069 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.076+0000 7fde2ea74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:16:09.069 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.076+0000 7fde2ea74640 1 -- 172.21.15.153:0/3223686157 >> v1:172.21.15.153:6790/0 conn(0x7fde280a9d70 legacy=0x7fde28140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:09.069 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.076+0000 7fde2ea74640 1 -- 172.21.15.153:0/3223686157 shutdown_connections 2024-03-27T03:16:09.069 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.076+0000 7fde2ea74640 1 -- 172.21.15.153:0/3223686157 >> 172.21.15.153:0/3223686157 conn(0x7fde28066a00 msgr2=0x7fde280b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:16:09.070 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.076+0000 7fde2ea74640 1 -- 172.21.15.153:0/3223686157 shutdown_connections 2024-03-27T03:16:09.070 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.076+0000 7fde2ea74640 1 -- 172.21.15.153:0/3223686157 wait complete. 2024-03-27T03:16:09.070 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:09.076+0000 7fde2ea74640 1 librados: shutdown 2024-03-27T03:16:09.086 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:16:15.893 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:16:16.041 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.047+0000 7f284c396640 10 monclient: get_monmap_and_config 2024-03-27T03:16:16.042 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.047+0000 7f284c396640 10 monclient: build_initial_monmap 2024-03-27T03:16:16.042 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.047+0000 7f284c396640 10 monclient: monmap: 2024-03-27T03:16:16.042 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:16:16.042 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:16.042 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:16:16.049367+0000 2024-03-27T03:16:16.042 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:16:16.049367+0000 2024-03-27T03:16:16.042 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:16:16.042 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:16.042 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:16:16.042 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:16:16.042 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:16:16.042 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:16.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.048+0000 7f284c396640 1 Processor -- start 2024-03-27T03:16:16.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.048+0000 7f284c396640 1 -- start start 2024-03-27T03:16:16.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.048+0000 7f284c396640 10 monclient: init 2024-03-27T03:16:16.043 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.049+0000 7f284c396640 10 monclient: _reopen_session rank -1 2024-03-27T03:16:16.047 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.049+0000 7f284c396640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:16:16.047 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.049+0000 7f284c396640 10 monclient(hunting): picked mon.noname-a con 0x7f28441457f0 addr v1:172.21.15.153:6789/0 2024-03-27T03:16:16.047 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.049+0000 7f284c396640 10 monclient(hunting): picked mon.noname-b con 0x7f284413f650 addr v1:172.21.15.153:6790/0 2024-03-27T03:16:16.047 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.049+0000 7f284c396640 10 monclient(hunting): picked mon.noname-c con 0x7f28440a9d70 addr v1:172.21.15.196:6789/0 2024-03-27T03:16:16.047 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.049+0000 7f284c396640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28440aaef0 con 0x7f28441457f0 2024-03-27T03:16:16.047 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.049+0000 7f284c396640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2844145bd0 con 0x7f28440a9d70 2024-03-27T03:16:16.047 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f284c396640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2844145eb0 con 0x7f284413f650 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f284c396640 10 monclient(hunting): _renew_subs 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f284c396640 10 monclient(hunting): authenticate will time out at 1936.356596s 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f284a933640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f28440a9d70 0x7f284414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.153:35046/0 (socket says 172.21.15.153:35046) 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f284a933640 1 -- 172.21.15.153:0/160713059 learned_addr learned my addr 172.21.15.153:0/160713059 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f2849130640 1 -- 172.21.15.153:0/160713059 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611787322 0 0) 0x7f2844145bd0 con 0x7f28440a9d70 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f2849130640 10 monclient(hunting): _negotiate 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f2849130640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f2849130640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f2849130640 10 monclient(hunting): my global_id is 4609 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f2849130640 1 -- 172.21.15.153:0/160713059 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2830003520 con 0x7f28440a9d70 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f2849130640 1 -- 172.21.15.153:0/160713059 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3017395896 0 0) 0x7f2830003520 con 0x7f28440a9d70 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f2849130640 1 -- 172.21.15.153:0/160713059 >> v1:172.21.15.153:6790/0 conn(0x7f284413f650 legacy=0x7f28440b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.050+0000 7f2849130640 1 -- 172.21.15.153:0/160713059 >> v1:172.21.15.153:6789/0 conn(0x7f28441457f0 legacy=0x7f284413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f2849130640 10 monclient: _finish_hunting 0 2024-03-27T03:16:16.048 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f2849130640 1 monclient: found mon.noname-c 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f2849130640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f2849130640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f2849130640 1 -- 172.21.15.153:0/160713059 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2844146340 con 0x7f28440a9d70 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f2849130640 10 monclient: _finish_auth 0 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f2849130640 10 monclient: _check_auth_tickets 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f2849130640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f2849130640 1 -- 172.21.15.153:0/160713059 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f283c001a10 con 0x7f28440a9d70 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f2849130640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f2849130640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f2849130640 10 monclient: dump: 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:16.049 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f284c396640 5 monclient: authenticate success, global_id 4609 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.051+0000 7f284c396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.052+0000 7f2849130640 1 -- 172.21.15.153:0/160713059 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f283c001bf0 con 0x7f28440a9d70 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.052+0000 7f2849130640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.052+0000 7f2849130640 1 -- 172.21.15.153:0/160713059 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f283c002000 con 0x7f28440a9d70 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.052+0000 7f2849130640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.052+0000 7f2849130640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.052+0000 7f2849130640 10 monclient: dump: 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:16.050 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.052+0000 7f284c396640 10 monclient: get_monmap_and_config success 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.052+0000 7f284c396640 4 set_mon_vals no callback set 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.053+0000 7f284c396640 10 monclient: shutdown 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.053+0000 7f284c396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.053+0000 7f284c396640 1 -- 172.21.15.153:0/160713059 >> v1:172.21.15.196:6789/0 conn(0x7f28440a9d70 legacy=0x7f284414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.053+0000 7f284c396640 1 -- 172.21.15.153:0/160713059 shutdown_connections 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.053+0000 7f284c396640 1 -- 172.21.15.153:0/160713059 >> 172.21.15.153:0/160713059 conn(0x7f2844066a00 msgr2=0x7f2844066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.053+0000 7f284c396640 1 -- 172.21.15.153:0/160713059 shutdown_connections 2024-03-27T03:16:16.051 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.053+0000 7f284c396640 1 -- 172.21.15.153:0/160713059 wait complete. 2024-03-27T03:16:16.058 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.054+0000 7f284c396640 10 monclient: build_initial_monmap 2024-03-27T03:16:16.058 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.054+0000 7f284c396640 10 monclient: monmap: 2024-03-27T03:16:16.058 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:16:16.058 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.055+0000 7f284c396640 1 librados: starting msgr at 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.055+0000 7f284c396640 1 librados: starting objecter 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.055+0000 7f284c396640 1 Processor -- start 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.055+0000 7f284c396640 1 -- start start 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.055+0000 7f284c396640 1 librados: setting wanted keys 2024-03-27T03:16:16.059 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.055+0000 7f284c396640 1 librados: calling monclient init 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.055+0000 7f284c396640 10 monclient: init 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.056+0000 7f284c396640 10 monclient: _reopen_session rank -1 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.056+0000 7f284c396640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.056+0000 7f284c396640 10 monclient(hunting): picked mon.noname-a con 0x7f28440a9d70 addr v1:172.21.15.153:6789/0 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.056+0000 7f284c396640 10 monclient(hunting): picked mon.noname-c con 0x7f284413f650 addr v1:172.21.15.153:6790/0 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.056+0000 7f284c396640 10 monclient(hunting): picked mon.noname-b con 0x7f28441457f0 addr v1:172.21.15.196:6789/0 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.056+0000 7f284c396640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2844142ab0 con 0x7f28440a9d70 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.056+0000 7f284c396640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284406fb50 con 0x7f28441457f0 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.056+0000 7f284c396640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284406e810 con 0x7f284413f650 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.056+0000 7f284c396640 10 monclient(hunting): _renew_subs 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.056+0000 7f284c396640 10 monclient(hunting): authenticate will time out at 1936.363163s 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.056+0000 7f2849931640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f284413f650 0x7f2844070250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:40804/0 (socket says 172.21.15.153:40804) 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.056+0000 7f2849931640 1 -- 172.21.15.153:0/366341742 learned_addr learned my addr 172.21.15.153:0/366341742 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.057+0000 7f283affd640 10 client.?.objecter ms_handle_connect 0x7f284413f650 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.057+0000 7f283affd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:16:16.060 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.057+0000 7f283affd640 10 client.?.objecter ms_handle_connect 0x7f28441457f0 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.057+0000 7f283affd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.057+0000 7f283affd640 10 client.?.objecter ms_handle_connect 0x7f28440a9d70 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.057+0000 7f283affd640 10 client.?.objecter resend_mon_ops 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.057+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407176783 0 0) 0x7f284406fb50 con 0x7f28441457f0 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.057+0000 7f283affd640 10 monclient(hunting): _negotiate 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.057+0000 7f283affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.057+0000 7f283affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.057+0000 7f283affd640 10 monclient(hunting): my global_id is 4612 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.057+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2828004eb0 con 0x7f28441457f0 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1312172212 0 0) 0x7f2828004eb0 con 0x7f28441457f0 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f284406fb50 con 0x7f28441457f0 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f283c001f00 con 0x7f28441457f0 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:16:16.061 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 10 monclient(hunting): dump: 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879807305 0 0) 0x7f284406e810 con 0x7f284413f650 2024-03-27T03:16:16.062 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 10 monclient(hunting): _negotiate 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 10 monclient(hunting): my global_id is 4667 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2828004eb0 con 0x7f284413f650 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944778138 0 0) 0x7f2844142ab0 con 0x7f28440a9d70 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 10 monclient(hunting): _negotiate 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 10 monclient(hunting): my global_id is 4581 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f284406e810 con 0x7f28440a9d70 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 609601165 0 0) 0x7f284406fb50 con 0x7f28441457f0 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.058+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 >> v1:172.21.15.153:6790/0 conn(0x7f284413f650 legacy=0x7f2844070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 >> v1:172.21.15.153:6789/0 conn(0x7f28440a9d70 legacy=0x7f284406e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f283affd640 10 monclient: _finish_hunting 0 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f283affd640 1 monclient: found mon.b 2024-03-27T03:16:16.063 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f283affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f283affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284406e9a0 con 0x7f28441457f0 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f283affd640 10 monclient: _finish_auth 0 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f283affd640 10 monclient: _check_auth_tickets 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f283affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f284c396640 5 monclient: authenticate success, global_id 4612 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f284c396640 10 monclient: _renew_subs 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f284c396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f284c396640 1 -- 172.21.15.153:0/366341742 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2844071d00 con 0x7f28441457f0 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f284c396640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f284c396640 10 monclient: _renew_subs 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f284c396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f284c396640 1 -- 172.21.15.153:0/366341742 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2844072000 con 0x7f28441457f0 2024-03-27T03:16:16.064 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.059+0000 7f284c396640 1 librados: init done 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.060+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f283c0020a0 con 0x7f28441457f0 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.060+0000 7f283affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.060+0000 7f284892f640 4 set_mon_vals no callback set 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.060+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f283c005110 con 0x7f28441457f0 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.060+0000 7f283affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.060+0000 7f283affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.060+0000 7f283affd640 10 monclient: dump: 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:16.065 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:16.066 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.061+0000 7f284c396640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:16.066 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.061+0000 7f284c396640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:16.066 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.061+0000 7f284c396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:16:16.066 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.061+0000 7f284c396640 1 -- 172.21.15.153:0/366341742 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f282c005080 con 0x7f28441457f0 2024-03-27T03:16:16.066 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.062+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f283c01b0e0 con 0x7f28441457f0 2024-03-27T03:16:16.066 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.062+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2380031179 0 0) 0x7f283c084f50 con 0x7f28441457f0 2024-03-27T03:16:16.066 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.062+0000 7f284a132640 10 monclient: get_auth_request con 0x7f282806de30 auth_method 0 2024-03-27T03:16:16.066 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.063+0000 7f283affd640 10 client.4612.objecter ms_dispatch 0x7f2844000b90 osd_map(11..11 src has 1..11) v4 2024-03-27T03:16:16.066 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.063+0000 7f283affd640 3 client.4612.objecter handle_osd_map got epochs [11,11] > 0 2024-03-27T03:16:16.066 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.063+0000 7f283affd640 3 client.4612.objecter handle_osd_map decoding full epoch 11 2024-03-27T03:16:16.066 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.063+0000 7f283affd640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-27T03:16:16.066 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.063+0000 7f283affd640 10 client.4612.objecter ms_handle_connect 0x7f282806de30 2024-03-27T03:16:16.067 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.072+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f283c050830 con 0x7f28441457f0 2024-03-27T03:16:16.067 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.072+0000 7f283affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:16.067 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.072+0000 7f283affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:16:16.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.425+0000 7f284c396640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:16.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.425+0000 7f284c396640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:16.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.425+0000 7f284c396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:16:16.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.425+0000 7f284c396640 1 -- 172.21.15.153:0/366341742 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f282c005330 con 0x7f28441457f0 2024-03-27T03:16:16.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.426+0000 7f283affd640 1 -- 172.21.15.153:0/366341742 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 992601923) 0x7f283c052760 con 0x7f28441457f0 2024-03-27T03:16:16.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.426+0000 7f283affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:16.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.426+0000 7f283affd640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:16:16.420 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:16:16.420 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":11,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:15:37.345901+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:15:37.345901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:16:16.423 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.428+0000 7f284c396640 10 librados: watch_flush enter 2024-03-27T03:16:16.423 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.429+0000 7f284c396640 10 librados: watch_flush exit 2024-03-27T03:16:16.423 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.429+0000 7f284c396640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-27T03:16:16.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.429+0000 7f284c396640 10 client.4612.objecter successfully canceled tick 2024-03-27T03:16:16.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.429+0000 7f284c396640 1 -- 172.21.15.153:0/366341742 >> v1:172.21.15.153:6816/2637937781 conn(0x7f282806de30 legacy=0x7f28280702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:16.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.429+0000 7f284c396640 10 monclient: shutdown 2024-03-27T03:16:16.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.429+0000 7f284c396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:16:16.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.429+0000 7f284c396640 1 -- 172.21.15.153:0/366341742 >> v1:172.21.15.196:6789/0 conn(0x7f28441457f0 legacy=0x7f2844070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:16.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.429+0000 7f284c396640 1 -- 172.21.15.153:0/366341742 shutdown_connections 2024-03-27T03:16:16.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.429+0000 7f284c396640 1 -- 172.21.15.153:0/366341742 >> 172.21.15.153:0/366341742 conn(0x7f2844066a00 msgr2=0x7f28440b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:16:16.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.429+0000 7f284c396640 1 -- 172.21.15.153:0/366341742 shutdown_connections 2024-03-27T03:16:16.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.429+0000 7f284c396640 1 -- 172.21.15.153:0/366341742 wait complete. 2024-03-27T03:16:16.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:16.429+0000 7f284c396640 1 librados: shutdown 2024-03-27T03:16:16.437 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:16:23.244 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:16:23.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.394+0000 7f3afb589640 10 monclient: get_monmap_and_config 2024-03-27T03:16:23.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.394+0000 7f3afb589640 10 monclient: build_initial_monmap 2024-03-27T03:16:23.387 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.394+0000 7f3afb589640 10 monclient: monmap: 2024-03-27T03:16:23.387 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:16:23.387 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:23.387 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:16:23.395232+0000 2024-03-27T03:16:23.388 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:16:23.395232+0000 2024-03-27T03:16:23.388 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:16:23.388 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:23.388 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:16:23.388 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:16:23.388 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:16:23.388 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:23.393 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.394+0000 7f3afb589640 1 Processor -- start 2024-03-27T03:16:23.393 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3afb589640 1 -- start start 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3afb589640 10 monclient: init 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3afb589640 10 monclient: _reopen_session rank -1 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3afb589640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3afb589640 10 monclient(hunting): picked mon.noname-b con 0x7f3af40b0500 addr v1:172.21.15.153:6790/0 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3afb589640 10 monclient(hunting): picked mon.noname-c con 0x7f3af40b08e0 addr v1:172.21.15.196:6789/0 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3afb589640 10 monclient(hunting): picked mon.noname-a con 0x7f3af40aeca0 addr v1:172.21.15.153:6789/0 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3afb589640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af4140620 con 0x7f3af40aeca0 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3afb589640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af413daf0 con 0x7f3af40b08e0 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3afb589640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40ae180 con 0x7f3af40b0500 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3afb589640 10 monclient(hunting): _renew_subs 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3afb589640 10 monclient(hunting): authenticate will time out at 1943.701307s 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3af9b26640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3af40aeca0 0x7f3af414b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39254/0 (socket says 172.21.15.153:39254) 2024-03-27T03:16:23.394 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.395+0000 7f3af9b26640 1 -- 172.21.15.153:0/3386825385 learned_addr learned my addr 172.21.15.153:0/3386825385 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.396+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874469467 0 0) 0x7f3af4140620 con 0x7f3af40aeca0 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.396+0000 7f3aebfff640 10 monclient(hunting): _negotiate 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.396+0000 7f3aebfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.396+0000 7f3aebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.396+0000 7f3aebfff640 10 monclient(hunting): my global_id is 4584 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.396+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae0003520 con 0x7f3af40aeca0 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.396+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116476868 0 0) 0x7f3af40ae180 con 0x7f3af40b0500 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.396+0000 7f3aebfff640 10 monclient(hunting): _negotiate 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.396+0000 7f3aebfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.396+0000 7f3aebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.396+0000 7f3aebfff640 10 monclient(hunting): my global_id is 4670 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.396+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af4140620 con 0x7f3af40b0500 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.397+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505464366 0 0) 0x7f3af413daf0 con 0x7f3af40b08e0 2024-03-27T03:16:23.395 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.397+0000 7f3aebfff640 10 monclient(hunting): _negotiate 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.397+0000 7f3aebfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.397+0000 7f3aebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.397+0000 7f3aebfff640 10 monclient(hunting): my global_id is 4615 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.397+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af40ae180 con 0x7f3af40b08e0 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.397+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1276103237 0 0) 0x7f3ae0003520 con 0x7f3af40aeca0 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.397+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 >> v1:172.21.15.153:6790/0 conn(0x7f3af40b0500 legacy=0x7f3af413ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.397+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 >> v1:172.21.15.196:6789/0 conn(0x7f3af40b08e0 legacy=0x7f3af413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3aebfff640 10 monclient: _finish_hunting 0 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3aebfff640 1 monclient: found mon.noname-a 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3aebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3aebfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.153:6789/0 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af40accf0 con 0x7f3af40aeca0 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3aebfff640 10 monclient: _finish_auth 0 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3aebfff640 10 monclient: _check_auth_tickets 2024-03-27T03:16:23.396 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3aebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3aec001a10 con 0x7f3af40aeca0 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3aebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3aebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3aebfff640 10 monclient: dump: 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:23.397 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3afb589640 5 monclient: authenticate success, global_id 4584 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.398+0000 7f3afb589640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.399+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3aec001bf0 con 0x7f3af40aeca0 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.399+0000 7f3aebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.399+0000 7f3aebfff640 1 -- 172.21.15.153:0/3386825385 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3aec002040 con 0x7f3af40aeca0 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.399+0000 7f3aebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.399+0000 7f3aebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.399+0000 7f3aebfff640 10 monclient: dump: 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:23.398 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:23.399 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:23.399 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:23.399 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:23.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.399+0000 7f3afb589640 10 monclient: get_monmap_and_config success 2024-03-27T03:16:23.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.399+0000 7f3afb589640 4 set_mon_vals no callback set 2024-03-27T03:16:23.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.400+0000 7f3afb589640 10 monclient: shutdown 2024-03-27T03:16:23.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.400+0000 7f3afb589640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:16:23.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.400+0000 7f3afb589640 1 -- 172.21.15.153:0/3386825385 >> v1:172.21.15.153:6789/0 conn(0x7f3af40aeca0 legacy=0x7f3af414b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:23.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.400+0000 7f3afb589640 1 -- 172.21.15.153:0/3386825385 shutdown_connections 2024-03-27T03:16:23.399 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.400+0000 7f3afb589640 1 -- 172.21.15.153:0/3386825385 >> 172.21.15.153:0/3386825385 conn(0x7f3af406b660 msgr2=0x7f3af406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.401+0000 7f3afb589640 1 -- 172.21.15.153:0/3386825385 shutdown_connections 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.401+0000 7f3afb589640 1 -- 172.21.15.153:0/3386825385 wait complete. 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.402+0000 7f3afb589640 10 monclient: build_initial_monmap 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.402+0000 7f3afb589640 10 monclient: monmap: 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:16:23.412 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.403+0000 7f3afb589640 1 librados: starting msgr at 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.403+0000 7f3afb589640 1 librados: starting objecter 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.403+0000 7f3afb589640 1 Processor -- start 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.403+0000 7f3afb589640 1 -- start start 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.403+0000 7f3afb589640 1 librados: setting wanted keys 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.403+0000 7f3afb589640 1 librados: calling monclient init 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.403+0000 7f3afb589640 10 monclient: init 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3afb589640 10 monclient: _reopen_session rank -1 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3afb589640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3afb589640 10 monclient(hunting): picked mon.noname-b con 0x7f3af40aeca0 addr v1:172.21.15.196:6789/0 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3afb589640 10 monclient(hunting): picked mon.noname-a con 0x7f3af40b0500 addr v1:172.21.15.153:6789/0 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3af8b24640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3af40b0500 0x7f3af413b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:40248/0 (socket says 172.21.15.153:40248) 2024-03-27T03:16:23.413 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3af8b24640 1 -- 172.21.15.153:0/604882514 learned_addr learned my addr 172.21.15.153:0/604882514 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3afb589640 10 monclient(hunting): picked mon.noname-c con 0x7f3af4070af0 addr v1:172.21.15.153:6790/0 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3afb589640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af413c780 con 0x7f3af40b0500 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3afb589640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af41d8620 con 0x7f3af40aeca0 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3afb589640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af41d87b0 con 0x7f3af4070af0 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3afb589640 10 monclient(hunting): _renew_subs 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3afb589640 10 monclient(hunting): authenticate will time out at 1943.711431s 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3ae9ffb640 10 client.?.objecter ms_handle_connect 0x7f3af40b0500 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.405+0000 7f3ae9ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 client.?.objecter ms_handle_connect 0x7f3af40aeca0 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 client.?.objecter ms_handle_connect 0x7f3af4070af0 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776582491 0 0) 0x7f3af41d8620 con 0x7f3af40aeca0 2024-03-27T03:16:23.414 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): my global_id is 4618 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad8004eb0 con 0x7f3af40aeca0 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980840949 0 0) 0x7f3af413c780 con 0x7f3af40b0500 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): my global_id is 4587 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af41d8620 con 0x7f3af40b0500 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420321083 0 0) 0x7f3af41d87b0 con 0x7f3af4070af0 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:23.415 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): my global_id is 4673 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af413c780 con 0x7f3af4070af0 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2570305851 0 0) 0x7f3ad8004eb0 con 0x7f3af40aeca0 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3af41d87b0 con 0x7f3af40aeca0 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3af00012b0 con 0x7f3af40aeca0 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): dump: 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:23.416 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.406+0000 7f3ae9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2075977510 0 0) 0x7f3af413c780 con 0x7f3af4070af0 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ad8004eb0 con 0x7f3af4070af0 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3aec001d60 con 0x7f3af4070af0 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 10 monclient(hunting): dump: 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:23.417 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3536813969 0 0) 0x7f3af41d87b0 con 0x7f3af40aeca0 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 >> v1:172.21.15.153:6790/0 conn(0x7f3af4070af0 legacy=0x7f3af413c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 >> v1:172.21.15.153:6789/0 conn(0x7f3af40b0500 legacy=0x7f3af413b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 1 monclient: found mon.b 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af413cef0 con 0x7f3af40aeca0 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 10 monclient: _finish_auth 0 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3ae9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:16:23.418 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3afb589640 5 monclient: authenticate success, global_id 4618 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3afb589640 10 monclient: _renew_subs 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3afb589640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3afb589640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af413d100 con 0x7f3af40aeca0 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3afb589640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3afb589640 10 monclient: _renew_subs 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3afb589640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3afb589640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af406fbe0 con 0x7f3af40aeca0 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.407+0000 7f3afb589640 1 librados: init done 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3af9325640 1 -- 172.21.15.153:0/604882514 >> v1:172.21.15.196:6789/0 conn(0x7f3af40aeca0 legacy=0x7f3af40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3af9325640 1 -- 172.21.15.153:0/604882514 >> v1:172.21.15.196:6789/0 conn(0x7f3af40aeca0 legacy=0x7f3af40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3af9325640 1 --1- 172.21.15.153:0/604882514 >> v1:172.21.15.196:6789/0 conn(0x7f3af40aeca0 0x7f3af40703e0 :-1 s=OPENED pgs=1335544 cs=1 l=1).handle_message read tag failed 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3af9325640 1 --1- 172.21.15.153:0/604882514 >> v1:172.21.15.196:6789/0 conn(0x7f3af40aeca0 0x7f3af40703e0 :-1 s=OPENED pgs=1335544 cs=1 l=1).fault on lossy channel, failing 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 monclient: ms_handle_reset current mon v1:172.21.15.196:6789/0 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 monclient: _reopen_session rank -1 2024-03-27T03:16:23.419 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 >> v1:172.21.15.196:6789/0 conn(0x7f3af40aeca0 legacy=0x7f3af40703e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 monclient(hunting): picked mon.a con 0x7f3af40b0500 addr v1:172.21.15.153:6789/0 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 monclient(hunting): picked mon.c con 0x7f3af4070af0 addr v1:172.21.15.153:6790/0 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 monclient(hunting): picked mon.b con 0x7f3ad800dca0 addr v1:172.21.15.196:6789/0 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3af41d87b0 con 0x7f3af40b0500 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3af413c780 con 0x7f3ad800dca0 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3ad8012de0 con 0x7f3af4070af0 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 monclient(hunting): _renew_subs 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 client.4618.objecter ms_handle_connect 0x7f3af40b0500 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 client.4618.objecter resend_mon_ops 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900435897 0 0) 0x7f3af41d87b0 con 0x7f3af40b0500 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:23.420 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f3ad80144d0 con 0x7f3af40b0500 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 client.4618.objecter ms_handle_connect 0x7f3af4070af0 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 client.4618.objecter resend_mon_ops 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 client.4618.objecter ms_handle_connect 0x7f3ad800dca0 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.408+0000 7f3ae9ffb640 10 client.4618.objecter resend_mon_ops 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115560459 0 0) 0x7f3af413c780 con 0x7f3ad800dca0 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3ae9ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3ae9ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.196:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f3af41d87b0 con 0x7f3ad800dca0 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3afb589640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3afb589640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155769551 0 0) 0x7f3ad8012de0 con 0x7f3af4070af0 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3ae9ffb640 10 monclient(hunting): _negotiate 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:23.421 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3ae9ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f3af413c780 con 0x7f3af4070af0 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2272162909 0 0) 0x7f3ad80144d0 con 0x7f3af40b0500 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.409+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 >> v1:172.21.15.153:6790/0 conn(0x7f3af4070af0 legacy=0x7f3ad800c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 >> v1:172.21.15.196:6789/0 conn(0x7f3ad800dca0 legacy=0x7f3ad8010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 10 monclient: _finish_hunting 0 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 1 monclient: found mon.a 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f3ad80141c0 con 0x7f3af40b0500 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af40ad0c0 con 0x7f3af40b0500 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:23.422 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af41d87b0 con 0x7f3af40b0500 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 10 monclient: _finish_auth 0 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 10 monclient: _check_auth_tickets 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3af0004c20 con 0x7f3af40b0500 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.410+0000 7f3ae9ffb640 10 monclient: dump: 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:23.423 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.411+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.0 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3af0004e40 con 0x7f3af40b0500 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.411+0000 7f3ae9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.411+0000 7f3aeb7fe640 4 set_mon_vals no callback set 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.414+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.0 v1:172.21.15.153:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f3af0004fb0 con 0x7f3af40b0500 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.414+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.0 v1:172.21.15.153:6789/0 6 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2380031179 0 0) 0x7f3af0084c60 con 0x7f3af40b0500 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.415+0000 7f3ae9ffb640 10 client.4618.objecter ms_dispatch 0x7f3af4000b90 osd_map(11..11 src has 1..11) v4 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.415+0000 7f3ae9ffb640 3 client.4618.objecter handle_osd_map got epochs [11,11] > 0 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.415+0000 7f3ae9ffb640 3 client.4618.objecter handle_osd_map decoding full epoch 11 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.415+0000 7f3ae9ffb640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.415+0000 7f3af8b24640 10 monclient: get_auth_request con 0x7f3af40aeca0 auth_method 0 2024-03-27T03:16:23.424 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.415+0000 7f3ae9ffb640 10 client.4618.objecter ms_handle_connect 0x7f3af40aeca0 2024-03-27T03:16:23.425 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.424+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3af0050540 con 0x7f3af40b0500 2024-03-27T03:16:23.425 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.424+0000 7f3ae9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:23.425 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.424+0000 7f3ae9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:16:23.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.434+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.0 v1:172.21.15.153:6789/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3af0052470 con 0x7f3af40b0500 2024-03-27T03:16:23.427 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.434+0000 7f3ae9ffb640 10 monclient: handle_mon_command_ack 1 not found 2024-03-27T03:16:23.770 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.777+0000 7f3afb589640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:23.770 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.777+0000 7f3afb589640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:23.770 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.777+0000 7f3afb589640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:23.770 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.777+0000 7f3afb589640 1 -- 172.21.15.153:0/604882514 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3af413d290 con 0x7f3af40b0500 2024-03-27T03:16:23.771 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.778+0000 7f3ae9ffb640 1 -- 172.21.15.153:0/604882514 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 992601923) 0x7f3af0053c90 con 0x7f3af40b0500 2024-03-27T03:16:23.771 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.778+0000 7f3ae9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:23.771 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.778+0000 7f3ae9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:16:23.771 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:16:23.771 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":11,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:15:37.345901+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:15:37.345901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:16:23.774 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.781+0000 7f3afb589640 10 librados: watch_flush enter 2024-03-27T03:16:23.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.781+0000 7f3afb589640 10 librados: watch_flush exit 2024-03-27T03:16:23.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.781+0000 7f3afb589640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-27T03:16:23.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.781+0000 7f3afb589640 10 client.4618.objecter successfully canceled tick 2024-03-27T03:16:23.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.781+0000 7f3afb589640 1 -- 172.21.15.153:0/604882514 >> v1:172.21.15.153:6816/2637937781 conn(0x7f3af40aeca0 legacy=0x7f3ad807baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:23.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.782+0000 7f3afb589640 10 monclient: shutdown 2024-03-27T03:16:23.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.782+0000 7f3afb589640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:16:23.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.782+0000 7f3afb589640 1 -- 172.21.15.153:0/604882514 >> v1:172.21.15.153:6789/0 conn(0x7f3af40b0500 legacy=0x7f3ad80056a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:23.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.782+0000 7f3afb589640 1 -- 172.21.15.153:0/604882514 shutdown_connections 2024-03-27T03:16:23.775 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.782+0000 7f3afb589640 1 -- 172.21.15.153:0/604882514 >> 172.21.15.153:0/604882514 conn(0x7f3af406b660 msgr2=0x7f3af40a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:16:23.776 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.782+0000 7f3afb589640 1 -- 172.21.15.153:0/604882514 shutdown_connections 2024-03-27T03:16:23.776 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.782+0000 7f3afb589640 1 -- 172.21.15.153:0/604882514 wait complete. 2024-03-27T03:16:23.776 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:23.782+0000 7f3afb589640 1 librados: shutdown 2024-03-27T03:16:23.791 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:16:30.598 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T03:16:30.751 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.758+0000 7f365e0e2640 10 monclient: get_monmap_and_config 2024-03-27T03:16:30.752 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.758+0000 7f365e0e2640 10 monclient: build_initial_monmap 2024-03-27T03:16:30.752 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.758+0000 7f365e0e2640 10 monclient: monmap: 2024-03-27T03:16:30.752 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:16:30.752 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:30.752 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:16:30.759267+0000 2024-03-27T03:16:30.752 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:16:30.759267+0000 2024-03-27T03:16:30.752 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:16:30.752 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:30.752 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:16:30.752 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-27T03:16:30.752 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-b 2024-03-27T03:16:30.752 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.759+0000 7f365e0e2640 1 Processor -- start 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.759+0000 7f365e0e2640 1 -- start start 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.759+0000 7f365e0e2640 10 monclient: init 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.760+0000 7f365e0e2640 10 monclient: _reopen_session rank -1 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.760+0000 7f365e0e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.760+0000 7f365e0e2640 10 monclient(hunting): picked mon.noname-c con 0x7f36580b0860 addr v1:172.21.15.196:6789/0 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.760+0000 7f365e0e2640 10 monclient(hunting): picked mon.noname-b con 0x7f365813ef30 addr v1:172.21.15.153:6790/0 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.760+0000 7f365e0e2640 10 monclient(hunting): picked mon.noname-a con 0x7f36580af8e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.760+0000 7f365e0e2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36580b0c40 con 0x7f36580af8e0 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.760+0000 7f365e0e2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36580aed10 con 0x7f36580b0860 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.760+0000 7f365e0e2640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36580afd50 con 0x7f365813ef30 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.760+0000 7f365e0e2640 10 monclient(hunting): _renew_subs 2024-03-27T03:16:30.756 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.760+0000 7f365e0e2640 10 monclient(hunting): authenticate will time out at 1951.066020s 2024-03-27T03:16:30.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.764+0000 7f3657fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f36580af8e0 0x7f3658149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:53037/0 (socket says 172.21.15.153:53037) 2024-03-27T03:16:30.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.764+0000 7f3657fff640 1 -- 172.21.15.153:0/3268861570 learned_addr learned my addr 172.21.15.153:0/3268861570 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:16:30.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 <== mon.2 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706788503 0 0) 0x7f36580afd50 con 0x7f365813ef30 2024-03-27T03:16:30.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 10 monclient(hunting): _negotiate 2024-03-27T03:16:30.760 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 10 monclient(hunting): my global_id is 4676 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3644003520 con 0x7f365813ef30 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29195780 0 0) 0x7f36580b0c40 con 0x7f36580af8e0 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 10 monclient(hunting): _negotiate 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 10 monclient(hunting): my global_id is 4590 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36580afd50 con 0x7f36580af8e0 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135741714 0 0) 0x7f36580aed10 con 0x7f36580b0860 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 10 monclient(hunting): _negotiate 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 10 monclient(hunting): my global_id is 4621 2024-03-27T03:16:30.761 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.767+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36580b0c40 con 0x7f36580b0860 2024-03-27T03:16:30.762 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 <== mon.2 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3516581116 0 0) 0x7f3644003520 con 0x7f365813ef30 2024-03-27T03:16:30.762 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 >> v1:172.21.15.196:6789/0 conn(0x7f36580b0860 legacy=0x7f365813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:30.762 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 >> v1:172.21.15.153:6789/0 conn(0x7f36580af8e0 legacy=0x7f3658149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:30.762 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 10 monclient: _finish_hunting 0 2024-03-27T03:16:30.762 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 1 monclient: found mon.noname-b 2024-03-27T03:16:30.762 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:16:30.762 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.153:6790/0 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36580ab1b0 con 0x7f365813ef30 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 10 monclient: _finish_auth 0 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 10 monclient: _check_auth_tickets 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 <== mon.2 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f36480015c0 con 0x7f365813ef30 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f36567fc640 10 monclient: dump: 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:30.763 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f365e0e2640 5 monclient: authenticate success, global_id 4676 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.768+0000 7f365e0e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.769+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 <== mon.2 v1:172.21.15.153:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3648001720 con 0x7f365813ef30 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.769+0000 7f36567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.769+0000 7f36567fc640 1 -- 172.21.15.153:0/3268861570 <== mon.2 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3648001b30 con 0x7f365813ef30 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.769+0000 7f36567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.769+0000 7f36567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.769+0000 7f36567fc640 10 monclient: dump: 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:30.764 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:30.765 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:30.765 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:30.765 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:30.765 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:30.765 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:30.765 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:30.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.769+0000 7f365e0e2640 10 monclient: get_monmap_and_config success 2024-03-27T03:16:30.765 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.769+0000 7f365e0e2640 4 set_mon_vals no callback set 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.770+0000 7f365e0e2640 10 monclient: shutdown 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.770+0000 7f365e0e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.770+0000 7f365e0e2640 1 -- 172.21.15.153:0/3268861570 >> v1:172.21.15.153:6790/0 conn(0x7f365813ef30 legacy=0x7f36580af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.770+0000 7f365e0e2640 1 -- 172.21.15.153:0/3268861570 shutdown_connections 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.770+0000 7f365e0e2640 1 -- 172.21.15.153:0/3268861570 >> 172.21.15.153:0/3268861570 conn(0x7f365806b660 msgr2=0x7f365806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.771+0000 7f365e0e2640 1 -- 172.21.15.153:0/3268861570 shutdown_connections 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.771+0000 7f365e0e2640 1 -- 172.21.15.153:0/3268861570 wait complete. 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.772+0000 7f365e0e2640 10 monclient: build_initial_monmap 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.772+0000 7f365e0e2640 10 monclient: monmap: 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-27T03:16:30.767 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:30.768 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a 2024-03-27T03:16:30.768 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-27T03:16:30.768 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.noname-c 2024-03-27T03:16:30.768 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:30.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.773+0000 7f365e0e2640 1 librados: starting msgr at 2024-03-27T03:16:30.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.773+0000 7f365e0e2640 1 librados: starting objecter 2024-03-27T03:16:30.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.773+0000 7f365e0e2640 1 Processor -- start 2024-03-27T03:16:30.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.773+0000 7f365e0e2640 1 -- start start 2024-03-27T03:16:30.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.773+0000 7f365e0e2640 1 librados: setting wanted keys 2024-03-27T03:16:30.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.773+0000 7f365e0e2640 1 librados: calling monclient init 2024-03-27T03:16:30.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.773+0000 7f365e0e2640 10 monclient: init 2024-03-27T03:16:30.768 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.774+0000 7f365e0e2640 10 monclient: _reopen_session rank -1 2024-03-27T03:16:30.769 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.774+0000 7f365e0e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T03:16:30.771 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.777+0000 7f365e0e2640 10 monclient(hunting): picked mon.noname-a con 0x7f36580af8e0 addr v1:172.21.15.153:6789/0 2024-03-27T03:16:30.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.777+0000 7f36577fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f36580af8e0 0x7f365812e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:53221/0 (socket says 172.21.15.153:53221) 2024-03-27T03:16:30.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.777+0000 7f36577fe640 1 -- 172.21.15.153:0/3171258528 learned_addr learned my addr 172.21.15.153:0/3171258528 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-27T03:16:30.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.778+0000 7f364ffff640 10 client.?.objecter ms_handle_connect 0x7f36580af8e0 2024-03-27T03:16:30.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.778+0000 7f364ffff640 10 client.?.objecter resend_mon_ops 2024-03-27T03:16:30.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.778+0000 7f365e0e2640 10 monclient(hunting): picked mon.noname-b con 0x7f36580b0860 addr v1:172.21.15.196:6789/0 2024-03-27T03:16:30.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.778+0000 7f365e0e2640 10 monclient(hunting): picked mon.noname-c con 0x7f365813ef30 addr v1:172.21.15.153:6790/0 2024-03-27T03:16:30.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.778+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365806e300 con 0x7f36580af8e0 2024-03-27T03:16:30.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.778+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36581d96c0 con 0x7f36580b0860 2024-03-27T03:16:30.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.778+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365806d9d0 con 0x7f365813ef30 2024-03-27T03:16:30.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.778+0000 7f365e0e2640 10 monclient(hunting): _renew_subs 2024-03-27T03:16:30.772 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.778+0000 7f365e0e2640 10 monclient(hunting): authenticate will time out at 1951.083951s 2024-03-27T03:16:30.779 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.785+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008444501 0 0) 0x7f365806e300 con 0x7f36580af8e0 2024-03-27T03:16:30.779 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.785+0000 7f364ffff640 10 monclient(hunting): _negotiate 2024-03-27T03:16:30.779 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.785+0000 7f364ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T03:16:30.779 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.785+0000 7f364ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T03:16:30.779 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.785+0000 7f364ffff640 10 monclient(hunting): my global_id is 4593 2024-03-27T03:16:30.779 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.785+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3638004eb0 con 0x7f36580af8e0 2024-03-27T03:16:30.779 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3076556065 0 0) 0x7f3638004eb0 con 0x7f36580af8e0 2024-03-27T03:16:30.779 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f365806e300 con 0x7f36580af8e0 2024-03-27T03:16:30.780 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3640001a40 con 0x7f36580af8e0 2024-03-27T03:16:30.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:30.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T03:16:30.789 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 10 monclient(hunting): dump: 2024-03-27T03:16:30.789 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:30.789 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:30.789 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:30.789 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 10 monclient(hunting): _finish_auth 0 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 <== mon.0 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 99227791 0 0) 0x7f365806e300 con 0x7f36580af8e0 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 >> v1:172.21.15.153:6790/0 conn(0x7f365813ef30 legacy=0x7f36581dfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 >> v1:172.21.15.196:6789/0 conn(0x7f36580b0860 legacy=0x7f36581df320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 10 monclient: _finish_hunting 0 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 1 monclient: found mon.a 2024-03-27T03:16:30.790 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365806db60 con 0x7f36580af8e0 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 10 monclient: _finish_auth 0 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 10 monclient: _check_auth_tickets 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f365e0e2640 5 monclient: authenticate success, global_id 4593 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f365e0e2640 10 monclient: _renew_subs 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f365e0e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36580703a0 con 0x7f36580af8e0 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 <== mon.0 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3640001ba0 con 0x7f36580af8e0 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f364ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.786+0000 7f3655ffb640 4 set_mon_vals no callback set 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.788+0000 7f365e0e2640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T03:16:30.791 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.788+0000 7f365e0e2640 10 monclient: _renew_subs 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.788+0000 7f365e0e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.788+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f365806e710 con 0x7f36580af8e0 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.788+0000 7f365e0e2640 1 librados: init done 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.791+0000 7f365e0e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.792+0000 7f365e0e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.792+0000 7f365e0e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.792+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365806fd10 con 0x7f36580af8e0 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.792+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 <== mon.0 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1996821508 0 0) 0x7f3640003a60 con 0x7f36580af8e0 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.792+0000 7f364ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.792+0000 7f364ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.792+0000 7f364ffff640 10 monclient: dump: 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:fsid 318002cf-3baf-46ae-b53f-8860af9c7a7a 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:30.792 INFO:teuthology.misc.health.smithi153.stderr:created 2024-03-27T03:04:54.165660+0000 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.a 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:2: v1:172.21.15.153:6790/0 mon.c 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr: 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.792+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 <== mon.0 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2320696779 0 0) 0x7f3640003c40 con 0x7f36580af8e0 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.793+0000 7f3656ffd640 10 monclient: get_auth_request con 0x7f363806dc00 auth_method 0 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.794+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 <== mon.0 v1:172.21.15.153:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2380031179 0 0) 0x7f3640054b30 con 0x7f36580af8e0 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.794+0000 7f364ffff640 10 client.4593.objecter ms_dispatch 0x7f3658000b90 osd_map(11..11 src has 1..11) v4 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.794+0000 7f364ffff640 3 client.4593.objecter handle_osd_map got epochs [11,11] > 0 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.794+0000 7f364ffff640 3 client.4593.objecter handle_osd_map decoding full epoch 11 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.794+0000 7f364ffff640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-27T03:16:30.793 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.794+0000 7f364ffff640 10 client.4593.objecter ms_handle_connect 0x7f363806dc00 2024-03-27T03:16:30.798 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.805+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 <== mon.0 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3640050640 con 0x7f36580af8e0 2024-03-27T03:16:30.798 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.805+0000 7f364ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T03:16:30.798 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:30.805+0000 7f364ffff640 10 monclient: _finish_command 1 = system:0 2024-03-27T03:16:31.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.115+0000 7f365e0e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:31.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.115+0000 7f365e0e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:31.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.115+0000 7f365e0e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.153:6789/0 2024-03-27T03:16:31.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.115+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f365806dcf0 con 0x7f36580af8e0 2024-03-27T03:16:31.109 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.117+0000 7f364ffff640 1 -- 172.21.15.153:0/3171258528 <== mon.0 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 992601923) 0x7f3640052570 con 0x7f36580af8e0 2024-03-27T03:16:31.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.117+0000 7f364ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T03:16:31.110 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.117+0000 7f364ffff640 10 monclient: _finish_command 2 = system:0 2024-03-27T03:16:31.113 INFO:teuthology.misc.health.smithi153.stdout: 2024-03-27T03:16:31.113 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":11,"fsid":"318002cf-3baf-46ae-b53f-8860af9c7a7a","created":"2024-03-27T03:05:32.440569+0000","modified":"2024-03-27T03:15:37.345901+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T03:15:37.345901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d21a393-d352-4f2e-bc5f-67ef3d73e26b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b1388fe9-4d83-4f69-a75a-3774e75e17c3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2193eae0-0507-4a25-b242-eb5d61931b56","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a8bacc46-d45a-463a-9b8e-c4af076e5a30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"125c9f30-749d-4e6a-8052-a3fcb44dfc43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c3395381-57c8-4362-9fe5-6d60312ce660","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddcca93d-aa6c-4c53-b3f3-7f49eef32854","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47e2cc28-454d-43f2-b42c-d3b267a87d2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T03:16:31.114 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.120+0000 7f365e0e2640 10 librados: watch_flush enter 2024-03-27T03:16:31.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.120+0000 7f365e0e2640 10 librados: watch_flush exit 2024-03-27T03:16:31.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.120+0000 7f365e0e2640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-27T03:16:31.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.120+0000 7f365e0e2640 10 client.4593.objecter successfully canceled tick 2024-03-27T03:16:31.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.121+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 >> v1:172.21.15.153:6816/2637937781 conn(0x7f363806dc00 legacy=0x7f36380700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:31.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.121+0000 7f365e0e2640 10 monclient: shutdown 2024-03-27T03:16:31.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.121+0000 7f365e0e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T03:16:31.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.121+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 >> v1:172.21.15.153:6789/0 conn(0x7f36580af8e0 legacy=0x7f365812e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T03:16:31.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.121+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 shutdown_connections 2024-03-27T03:16:31.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.121+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 >> 172.21.15.153:0/3171258528 conn(0x7f365806b660 msgr2=0x7f36580adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T03:16:31.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.121+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 shutdown_connections 2024-03-27T03:16:31.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.121+0000 7f365e0e2640 1 -- 172.21.15.153:0/3171258528 wait complete. 2024-03-27T03:16:31.115 INFO:teuthology.misc.health.smithi153.stderr:2024-03-27T03:16:31.121+0000 7f365e0e2640 1 librados: shutdown 2024-03-27T03:16:31.132 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T03:16:31.132 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-27T03:16:31.152 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-27T03:16:31.152 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-27T03:16:31.152 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.204 INFO:tasks.ceph.osd.0:Stopped 2024-03-27T03:16:31.204 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-27T03:16:31.204 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.250 INFO:tasks.ceph.osd.1:Stopped 2024-03-27T03:16:31.250 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-27T03:16:31.250 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.285 INFO:tasks.ceph.osd.2:Stopped 2024-03-27T03:16:31.285 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-27T03:16:31.286 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.359 INFO:tasks.ceph.osd.3:Stopped 2024-03-27T03:16:31.359 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-27T03:16:31.359 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.398 INFO:tasks.ceph.osd.4:Stopped 2024-03-27T03:16:31.399 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-27T03:16:31.399 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.478 INFO:tasks.ceph.osd.5:Stopped 2024-03-27T03:16:31.478 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-27T03:16:31.478 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.559 INFO:tasks.ceph.osd.6:Stopped 2024-03-27T03:16:31.560 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-27T03:16:31.560 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.593 INFO:tasks.ceph.osd.7:Stopped 2024-03-27T03:16:31.593 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-27T03:16:31.593 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-27T03:16:31.594 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.619 INFO:tasks.ceph.mgr.y:Stopped 2024-03-27T03:16:31.619 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-27T03:16:31.619 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.659 INFO:tasks.ceph.mgr.x:Stopped 2024-03-27T03:16:31.659 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-27T03:16:31.659 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-27T03:16:31.659 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.670 INFO:tasks.ceph.mon.a:Stopped 2024-03-27T03:16:31.671 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-27T03:16:31.671 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.723 INFO:tasks.ceph.mon.c:Stopped 2024-03-27T03:16:31.723 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-27T03:16:31.723 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T03:16:31.734 INFO:tasks.ceph.mon.b:Stopped 2024-03-27T03:16:31.734 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-27T03:16:31.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-27T03:16:31.794 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-27T03:16:31.794 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-27T03:16:33.733 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-27T03:16:33.733 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-27T03:16:33.838 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-27T03:16:33.838 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-27T03:16:33.905 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-27T03:16:33.906 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-27T03:16:33.962 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-27T03:16:33.962 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-27T03:16:37.347 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-27T03:16:37.348 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-27T03:16:37.410 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-27T03:16:37.410 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-27T03:16:37.483 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-27T03:16:37.484 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-27T03:16:37.553 INFO:tasks.ceph:Archiving mon data... 2024-03-27T03:16:37.555 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698/data/mon.a.tgz 2024-03-27T03:16:37.555 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-27T03:16:37.573 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.fE2EV8NJAN 2024-03-27T03:16:37.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.fE2EV8NJAN 2024-03-27T03:16:37.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.fE2EV8NJAN 2024-03-27T03:16:37.776 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.fE2EV8NJAN is 127KB 2024-03-27T03:16:37.802 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.fE2EV8NJAN 2024-03-27T03:16:37.816 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698/data/mon.c.tgz 2024-03-27T03:16:37.817 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-27T03:16:37.871 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.GiayQkM4C7 2024-03-27T03:16:37.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.GiayQkM4C7 2024-03-27T03:16:37.959 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.GiayQkM4C7 2024-03-27T03:16:38.043 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.GiayQkM4C7 is 127KB 2024-03-27T03:16:38.065 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.GiayQkM4C7 2024-03-27T03:16:38.079 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698/data/mon.b.tgz 2024-03-27T03:16:38.079 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-03-27T03:16:38.097 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.Cd3LmWkGU9 2024-03-27T03:16:38.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Cd3LmWkGU9 2024-03-27T03:16:38.204 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.Cd3LmWkGU9 2024-03-27T03:16:38.291 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.Cd3LmWkGU9 is 127KB 2024-03-27T03:16:38.342 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.Cd3LmWkGU9 2024-03-27T03:16:38.357 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-27T03:16:38.357 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-27T03:16:38.360 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-27T03:16:38.439 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-27T03:16:38.480 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-27T03:16:38.480 INFO:tasks.ceph:Archiving crash dumps... 2024-03-27T03:16:38.482 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698/remote/smithi153/crash 2024-03-27T03:16:38.483 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-27T03:16:38.509 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698/remote/smithi196/crash 2024-03-27T03:16:38.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-27T03:16:38.547 INFO:tasks.ceph:Compressing logs... 2024-03-27T03:16:38.547 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-27T03:16:38.554 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-27T03:16:38.619 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28818.log 2024-03-27T03:16:38.619 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41556.log 2024-03-27T03:16:38.619 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.tmp-client.admin.28818.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-27T03:16:38.619 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28818.log.gz 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41165.log 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38782.log 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41556.log.gz 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41634.log 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40892.log 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41165.log.gz 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39094.log 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38782.log.gz 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40892.log: gzip -5 86.8% --verbose -- replaced with /var/log/ceph/ceph-client.admin.41634.log.gz -- 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr: /var/log/ceph/ceph-client.admin.39601.log 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39484.log 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40892.log.gz 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41243.log 2024-03-27T03:16:38.620 INFO:teuthology.orchestra.run.smithi153.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39094.log.gz 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41399.log 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39601.log.gz 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41243.log: 86.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.39484.log.gz -5 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41009.log 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40306.log 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41243.log.gz 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39718.log 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41399.log.gz 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39757.log 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41009.log.gz 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39718.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40306.log.gz 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39211.log 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39718.log.gz 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41360.log 2024-03-27T03:16:38.621 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40619.log 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39757.log.gz 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39016.log 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39211.log.gz 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40424.log 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41360.log.gz 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39016.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40619.log.gz 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40189.log 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40424.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39016.log.gz 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39562.log 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41673.log 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40424.log.gz 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39562.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40189.log.gz 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40736.log 2024-03-27T03:16:38.622 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40931.log 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39562.log.gz 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40541.log 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41673.log.gz 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38821.log 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40736.log.gz 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40931.log.gz 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39367.log 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38351.log 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40541.log.gz 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40580.log 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38821.log.gz 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38351.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39367.log.gz 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-27T03:16:38.623 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40345.log 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38351.log.gz 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41204.log 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40580.log.gz 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40267.log 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz/var/log/ceph/ceph-client.admin.41204.log: 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38312.log 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40345.log.gz 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38626.log 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41204.log.gz 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40267.log.gz 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40072.log 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38704.log 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38312.log.gz 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39250.log 2024-03-27T03:16:38.624 INFO:teuthology.orchestra.run.smithi153.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38626.log.gz 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38860.log 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38704.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40072.log.gz 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41478.log 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38704.log.gz 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38860.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39250.log.gz 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41595.log 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41478.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38860.log.gz 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41087.log 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41478.log.gz 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38899.log 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41087.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41595.log.gz 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39955.log 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39328.log 2024-03-27T03:16:38.625 INFO:teuthology.orchestra.run.smithi153.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41087.log.gz 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38899.log: /var/log/ceph/ceph-client.admin.39955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40970.log 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39328.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38899.log.gz 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41048.log 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39955.log.gz 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38743.log 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39328.log.gz 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39994.log 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40970.log.gz 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39835.log 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41048.log.gz 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38468.log 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38743.log.gz 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39835.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39994.log.gz 2024-03-27T03:16:38.626 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38468.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39835.log.gz 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39640.log 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40775.log 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.3.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38468.log.gz 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39874.log 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38665.log 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39640.log.gz 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39055.log 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39874.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40775.log.gz 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39679.log 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39874.log.gz 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38390.log 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38665.log.gz 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41439.log 2024-03-27T03:16:38.627 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39679.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39055.log.gz 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38390.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41439.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39679.log.gz 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr:gzip 86.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.38390.log.gz 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40111.log 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41439.log.gz 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38938.log 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39133.log 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40111.log: /var/log/ceph/ceph-client.admin.38938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39796.log 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40111.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40697.log 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38938.log.gz 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39133.log: /var/log/ceph/ceph-client.admin.39796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34637.log 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr: 86.6%/var/log/ceph/ceph-client.admin.40697.log: -- replaced with /var/log/ceph/ceph-client.admin.39133.log.gz 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39796.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41126.log 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41712.log 2024-03-27T03:16:38.628 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40697.log.gz 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41517.log 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34637.log.gz 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41126.log.gz 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41712.log: /var/log/ceph/ceph-client.admin.41517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39445.log 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41282.log 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41517.log.gz 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40150.log 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41282.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39445.log.gz 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41712.log.gz 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40502.log 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41282.log.gz 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40150.log: -- /var/log/ceph/ceph.log 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39172.log 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40150.log.gz 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40502.log.gz 2024-03-27T03:16:38.629 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40228.log 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.2.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39172.log.gz 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34964.log 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph-client.admin.40228.log: 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41321.log 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40228.log.gz 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40033.log 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34964.log: /var/log/ceph/ceph-client.admin.41321.log: 87.1% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34964.log.gz 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41321.log.gz 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39523.log 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40384.log 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40033.log.gz 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38429.log 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40384.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39523.log.gz 2024-03-27T03:16:38.630 INFO:teuthology.orchestra.run.smithi153.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40384.log.gz 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38546.log 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41751.log 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38429.log: /var/log/ceph/ceph-client.admin.38546.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38429.log.gz 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.38546.log.gz 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35033.log 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39916.log 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41751.log: /var/log/ceph/ceph-client.admin.35033.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41751.log.gz 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35033.log.gz 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40814.log 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39916.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39916.log.gz 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39289.log 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39406.log 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40814.log: /var/log/ceph/ceph-client.admin.39289.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40814.log.gz 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39289.log.gz 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40463.log 2024-03-27T03:16:38.631 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39406.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39406.log.gz 2024-03-27T03:16:38.632 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38587.log 2024-03-27T03:16:38.632 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40853.log 2024-03-27T03:16:38.632 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40463.log: /var/log/ceph/ceph-client.admin.38587.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40463.log.gz 2024-03-27T03:16:38.632 INFO:teuthology.orchestra.run.smithi153.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38587.log.gz 2024-03-27T03:16:38.632 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40658.log 2024-03-27T03:16:38.632 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38507.log 2024-03-27T03:16:38.632 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40853.log: /var/log/ceph/ceph-client.admin.40658.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40853.log.gz 2024-03-27T03:16:38.632 INFO:teuthology.orchestra.run.smithi153.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40658.log.gz 2024-03-27T03:16:38.632 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38977.log 2024-03-27T03:16:38.632 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-27T03:16:38.632 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38507.log: /var/log/ceph/ceph-client.admin.38977.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38507.log.gz 2024-03-27T03:16:38.632 INFO:teuthology.orchestra.run.smithi153.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38977.log.gz 2024-03-27T03:16:38.637 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-27T03:16:38.637 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34677.log 2024-03-27T03:16:38.637 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34797.log 2024-03-27T03:16:38.637 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34677.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34797.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34677.log.gz 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph.audit.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34797.log.gz 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34877.log 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34637.log 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34837.log 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34637.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34757.log 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34877.log.gz 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr: 87.9%/var/log/ceph/ceph-client.admin.34837.log: -- replaced with /var/log/ceph/ceph-client.admin.34637.log.gzgzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34757.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34837.log.gz 2024-03-27T03:16:38.638 INFO:teuthology.orchestra.run.smithi196.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34757.log.gz 2024-03-27T03:16:38.639 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-27T03:16:38.639 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34717.log 2024-03-27T03:16:38.639 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34597.log 2024-03-27T03:16:38.639 INFO:teuthology.orchestra.run.smithi196.stderr: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-27T03:16:38.639 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34717.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34717.log.gz 2024-03-27T03:16:38.639 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34597.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34597.log.gz 2024-03-27T03:16:38.639 INFO:teuthology.orchestra.run.smithi196.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-27T03:16:38.668 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.c.log: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-27T03:16:49.774 INFO:teuthology.orchestra.run.smithi196.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-27T03:16:49.998 INFO:teuthology.orchestra.run.smithi153.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-27T03:16:50.044 INFO:teuthology.orchestra.run.smithi196.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-27T03:16:50.687 INFO:teuthology.orchestra.run.smithi196.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-27T03:16:54.101 INFO:teuthology.orchestra.run.smithi153.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-27T03:16:54.107 INFO:teuthology.orchestra.run.smithi196.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-27T03:16:54.162 INFO:teuthology.orchestra.run.smithi153.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-27T03:16:55.712 INFO:teuthology.orchestra.run.smithi153.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-27T03:17:24.024 INFO:teuthology.orchestra.run.smithi153.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-27T03:17:31.078 INFO:teuthology.orchestra.run.smithi153.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-27T03:17:31.080 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-27T03:17:31.080 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m52.514s 2024-03-27T03:17:31.080 INFO:teuthology.orchestra.run.smithi153.stderr:user 2m31.807s 2024-03-27T03:17:31.080 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m5.543s 2024-03-27T03:17:49.414 INFO:teuthology.orchestra.run.smithi196.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-27T03:17:49.416 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-27T03:17:49.416 INFO:teuthology.orchestra.run.smithi196.stderr:real 1m10.811s 2024-03-27T03:17:49.416 INFO:teuthology.orchestra.run.smithi196.stderr:user 1m56.481s 2024-03-27T03:17:49.416 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m4.246s 2024-03-27T03:17:49.417 INFO:tasks.ceph:Archiving logs... 2024-03-27T03:17:49.417 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698/remote/smithi153/log 2024-03-27T03:17:49.419 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-27T03:18:11.765 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698/remote/smithi196/log 2024-03-27T03:18:11.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-27T03:18:30.927 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-27T03:18:31.167 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e8cc2d8ffe8948c5b6317ded90f45368 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-27T03:18:31.173 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-27T03:18:31.189 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-27T03:18:31.231 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-27T03:18:31.231 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-27T03:18:31.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-27T03:18:31.329 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-27T03:18:31.341 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-27T03:18:31.341 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T03:18:31.343 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T03:18:31.357 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-27T03:18:31.359 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-27T03:18:31.362 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T03:18:31.362 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-27T03:18:31.362 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 52 +388us[ +386us] +/- 86ms 2024-03-27T03:18:31.362 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -535us[ -537us] +/- 66ms 2024-03-27T03:18:31.362 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +269us[ +267us] +/- 76ms 2024-03-27T03:18:31.362 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T03:18:31.365 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T03:18:31.365 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-27T03:18:31.365 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +381us[ +381us] +/- 86ms 2024-03-27T03:18:31.365 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -523us[ -525us] +/- 66ms 2024-03-27T03:18:31.365 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +270us[ +270us] +/- 76ms 2024-03-27T03:18:31.365 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T03:18:31.366 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-27T03:18:31.378 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-27T03:18:31.378 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-27T03:18:31.405 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-27T03:18:31.474 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-27T03:18:31.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-27T03:18:31.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T03:18:31.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-27T03:18:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T03:18:31.617 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-27T03:18:31.690 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-27T03:18:31.719 INFO:teuthology.task.internal:Duration was 1236.106712 seconds 2024-03-27T03:18:31.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-27T03:18:31.750 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-27T03:18:31.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-27T03:18:31.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-27T03:18:31.801 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T03:18:31.803 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T03:18:32.108 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-27T03:18:32.108 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-03-27T03:18:32.108 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-27T03:18:32.133 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-03-27T03:18:32.133 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-27T03:18:32.158 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-27T03:18:32.159 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-27T03:18:32.178 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-27T03:18:32.231 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-27T03:18:32.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-27T03:18:32.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-27T03:18:32.370 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-27T03:18:32.571 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-27T03:18:32.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-27T03:18:32.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-27T03:18:32.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-27T03:18:32.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-27T03:18:32.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-27T03:18:32.674 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-03-27T03:18:32.693 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-03-27T03:18:32.718 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-27T03:18:32.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T03:18:32.752 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-27T03:18:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T03:18:32.773 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-27T03:18:32.783 INFO:teuthology.task.internal:Transferring archived files... 2024-03-27T03:18:32.784 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698/remote/smithi153 2024-03-27T03:18:32.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-27T03:18:32.828 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698/remote/smithi196 2024-03-27T03:18:32.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-27T03:18:32.863 INFO:teuthology.task.internal:Removing archive directory... 2024-03-27T03:18:32.863 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-27T03:18:32.866 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-27T03:18:32.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-27T03:18:32.924 INFO:teuthology.task.internal:Not uploading archives. 2024-03-27T03:18:32.924 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-27T03:18:32.973 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-27T03:18:32.973 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-27T03:18:32.975 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-27T03:18:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 03:18 /home/ubuntu/cephtest 2024-03-27T03:18:32.989 INFO:teuthology.orchestra.run.smithi196.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 03:18 /home/ubuntu/cephtest 2024-03-27T03:18:32.991 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-27T03:18:33.011 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-27T03:18:33.079 INFO:teuthology.nuke:Checking targets against current locks 2024-03-27T03:18:33.100 DEBUG:teuthology.nuke:shortname: smithi153 2024-03-27T03:18:33.101 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T03:18:33.122 DEBUG:teuthology.nuke:shortname: smithi196 2024-03-27T03:18:33.122 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T03:18:33.126 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 02:41:05.090063', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCE2dRf1adqYcZW5xkSEOuI5DfuEsx1HXm8zZPbhMJqUbT0jV1f7DJMaWOlAr3wqgK+GdDh9ylXyGJ1Xn1vnopk='} 2024-03-27T03:18:33.145 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622698', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 02:41:05.092037', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFZ5kbULDe0hPvOIpHU+FY7udHbBIdr9lbE57zrSjNk6OqDJDxHa8jlUaa1AYKifJLnaUM/ut1vO0rQodJQVeaw='} 2024-03-27T03:18:33.176 INFO:teuthology.orchestra.console.smithi153:Power off 2024-03-27T03:18:33.177 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-27T03:18:33.199 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-03-27T03:18:33.200 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T03:18:33.215 INFO:teuthology.orchestra.console.smithi196:Power off 2024-03-27T03:18:33.216 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-27T03:18:33.225 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-27T03:18:33.237 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off 2024-03-27T03:18:33.238 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T03:18:33.261 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-27T03:18:37.226 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T03:18:37.262 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T03:18:37.339 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-27T03:18:37.373 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-27T03:18:41.340 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T03:18:41.374 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T03:18:41.451 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-27T03:18:41.484 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-27T03:18:45.453 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T03:18:45.485 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T03:18:45.563 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-03-27T03:18:45.596 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off 2024-03-27T03:18:45.664 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-03-27T03:18:45.697 INFO:teuthology.orchestra.console.smithi196:Power off completed 2024-03-27T03:18:45.801 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1236.1067123413086 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e8cc2d8ffe8948c5b6317ded90f45368 status: fail success: false 2024-03-27T03:18:45.801 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T03:18:45.875 INFO:teuthology.run:FAIL